[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:21 dist-test-slave-dmhs krb5kdc[9274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:21 dist-test-slave-dmhs krb5kdc[9274](info): set up 2 sockets
Sep 28 15:02:21 dist-test-slave-dmhs krb5kdc[9274](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 15:02:22.849284 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:22 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:23.002132 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:41037
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:41037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:23.340288 9287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:23.340636 9287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:23.340740 9287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:23.348636 9287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:23.348758 9287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:23.348891 9287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:23.349053 9287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:23.349136 9287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:23.359822 9287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:41037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:41037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:23.361207 9287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:23.362473 9287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:23.375757 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:23.375895 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:23.375906 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:23.376958 9287 server_base.cc:1034] running on GCE node
I20260928 15:02:23.377719 9287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:23.379168 9287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:23.380368 9287 hybrid_clock.cc:648] HybridClock initialized: now 1790607743380309 us; error 79 us; skew 500 ppm
I20260928 15:02:23.380708 9287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:23 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:23.384127 9287 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:23.385710 9287 webserver.cc:460] Webserver started at http://127.9.12.254:35105/ using document root <none> and password file <none>
I20260928 15:02:23.386354 9287 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:23.386456 9287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:23.386695 9287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:23.389667 9287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "3b520b1b5fd94ed1affda751bc6f534f"
format_stamp: "Formatted at 2026-09-28 15:02:23 on dist-test-slave-dmhs"
server_key: "3ce6dd2f718da9a27607028bf84005d1"
server_key_iv: "4a4c955d92f46282731e846565aed32a"
server_key_version: "encryptionkey@0"
I20260928 15:02:23.390378 9287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "3b520b1b5fd94ed1affda751bc6f534f"
format_stamp: "Formatted at 2026-09-28 15:02:23 on dist-test-slave-dmhs"
server_key: "3ce6dd2f718da9a27607028bf84005d1"
server_key_iv: "4a4c955d92f46282731e846565aed32a"
server_key_version: "encryptionkey@0"
I20260928 15:02:23.395339 9287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 15:02:23.398648 9302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:23.400022 9287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:23.400151 9287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "3b520b1b5fd94ed1affda751bc6f534f"
format_stamp: "Formatted at 2026-09-28 15:02:23 on dist-test-slave-dmhs"
server_key: "3ce6dd2f718da9a27607028bf84005d1"
server_key_iv: "4a4c955d92f46282731e846565aed32a"
server_key_version: "encryptionkey@0"
I20260928 15:02:23.400368 9287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:23.424587 9287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:23.429209 9287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:23.429433 9287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:23.451149 9287 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:41037
I20260928 15:02:23.451185 9353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:41037 every 8 connection(s)
I20260928 15:02:23.452508 9287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:23.456205 9354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:23.456976 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 9287
I20260928 15:02:23.457270 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:23.459052 9267 external_mini_cluster.cc:1428] Setting key 16ccf7055ba783885c2d28a1d26a2ffb
I20260928 15:02:23.466727 9354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f: Bootstrap starting.
I20260928 15:02:23.469847 9354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:23.471153 9354 log.cc:824] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:23.474502 9354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f: No bootstrap required, opened a new log
I20260928 15:02:23.483808 9354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b520b1b5fd94ed1affda751bc6f534f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41037 } }
I20260928 15:02:23.484339 9354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:23.484442 9354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b520b1b5fd94ed1affda751bc6f534f, State: Initialized, Role: FOLLOWER
I20260928 15:02:23.485010 9354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b520b1b5fd94ed1affda751bc6f534f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41037 } }
I20260928 15:02:23.485237 9354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:23.485394 9354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:23.485512 9354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:23.488471 9354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b520b1b5fd94ed1affda751bc6f534f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41037 } }
I20260928 15:02:23.489105 9354 leader_election.cc:302] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b520b1b5fd94ed1affda751bc6f534f; no voters:
I20260928 15:02:23.489722 9354 leader_election.cc:288] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:23.489890 9359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:23.490752 9359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [term 1 LEADER]: Becoming Leader. State: Replica: 3b520b1b5fd94ed1affda751bc6f534f, State: Running, Role: LEADER
I20260928 15:02:23.491281 9359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b520b1b5fd94ed1affda751bc6f534f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41037 } }
I20260928 15:02:23.491999 9354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:23.494607 9361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b520b1b5fd94ed1affda751bc6f534f. Latest consensus state: current_term: 1 leader_uuid: "3b520b1b5fd94ed1affda751bc6f534f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b520b1b5fd94ed1affda751bc6f534f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41037 } } }
I20260928 15:02:23.495282 9361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:23.496140 9360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b520b1b5fd94ed1affda751bc6f534f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b520b1b5fd94ed1affda751bc6f534f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41037 } } }
I20260928 15:02:23.496534 9360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f [sys.catalog]: This master's current role is: LEADER
Sep 28 15:02:23 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:23.506004 9364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:23.511688 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:23.462743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.9.12.254:41037)
0928 15:02:23.463608 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.463636 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.464937 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 15:02:23.475232 (+ 10295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.475258 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.475727 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.476202 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:23.478963 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.481117 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.482719 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.483765 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.483940 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:23.499010 (+ 15070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:23.499026 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:23.499051 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:23.499119 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:23.504408 (+ 5289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:23.505834 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:23.505841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:23.505848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:23.506118 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:23.506735 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:23.506741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:23.506747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:23.507521 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:23.507851 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:23.509261 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0928 15:02:23.509703 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260928 15:02:23.513161 9364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:23.526540 9364 catalog_manager.cc:1347] Generated new cluster ID: c0feb347773141349f0ec57e1dd10627
I20260928 15:02:23.526647 9364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:23.544800 9364 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:23.546581 9364 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:23.553834 9364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b520b1b5fd94ed1affda751bc6f534f: Generated new TSK 0
I20260928 15:02:23.554651 9364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:23.602444 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41037
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:23.960419 9380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:23.960737 9380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:23.960933 9380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:23.967907 9380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:23.968057 9380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:23.968255 9380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:23.977492 9380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:23.978780 9380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:23.980136 9380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:23.994336 9386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:23.994426 9388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:23.994378 9385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:23.995150 9380 server_base.cc:1034] running on GCE node
I20260928 15:02:23.995790 9380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:23.996820 9380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:23.998059 9380 hybrid_clock.cc:648] HybridClock initialized: now 1790607743998009 us; error 40 us; skew 500 ppm
I20260928 15:02:23.998383 9380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:23 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:24.001614 9380 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:24.002916 9380 webserver.cc:460] Webserver started at http://127.9.12.193:34245/ using document root <none> and password file <none>
I20260928 15:02:24.003575 9380 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:24.003671 9380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:24.004016 9380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:24.007138 9380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "b61078b96bd24f6cb4c080ff4c009f29"
format_stamp: "Formatted at 2026-09-28 15:02:24 on dist-test-slave-dmhs"
server_key: "890c65181398b8fc8e3848beb001e9ce"
server_key_iv: "935c66f005d4b32dece6d327e1549f0b"
server_key_version: "encryptionkey@0"
I20260928 15:02:24.007951 9380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "b61078b96bd24f6cb4c080ff4c009f29"
format_stamp: "Formatted at 2026-09-28 15:02:24 on dist-test-slave-dmhs"
server_key: "890c65181398b8fc8e3848beb001e9ce"
server_key_iv: "935c66f005d4b32dece6d327e1549f0b"
server_key_version: "encryptionkey@0"
I20260928 15:02:24.013458 9380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:02:24.016829 9395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:24.018098 9380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:24.018283 9380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "b61078b96bd24f6cb4c080ff4c009f29"
format_stamp: "Formatted at 2026-09-28 15:02:24 on dist-test-slave-dmhs"
server_key: "890c65181398b8fc8e3848beb001e9ce"
server_key_iv: "935c66f005d4b32dece6d327e1549f0b"
server_key_version: "encryptionkey@0"
I20260928 15:02:24.018486 9380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:24.039408 9380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:24.043726 9380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:24.043990 9380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:24.045022 9380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:24.046242 9380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:24.046324 9380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:24.046404 9380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:24.046446 9380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:24.083787 9380 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:39637
I20260928 15:02:24.084057 9507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:39637 every 8 connection(s)
I20260928 15:02:24.085220 9380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:24.089470 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 9380
I20260928 15:02:24.089723 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:24.090368 9267 external_mini_cluster.cc:1428] Setting key a3264f3239b292d6a41262949a2bc3e4
Sep 28 15:02:24 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:24.107224 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:24.087795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:49753 (local address 127.9.12.254:41037)
0928 15:02:24.088414 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:24.088419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:24.089175 (+ 756us) server_negotiation.cc:407] Connection header received
0928 15:02:24.091233 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:24.091239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:24.091332 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:24.091453 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:24.093134 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.093849 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.095390 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.095829 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.095962 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:24.099456 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:24.099465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:24.099473 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:24.099519 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:24.103786 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:24.104817 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:24.104824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:24.104830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:24.104995 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:24.105451 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:24.105456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:24.105465 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:24.105725 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:24.105900 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:24.106611 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 15:02:24.106829 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":399,"threads_started":1}
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:02:24.108354 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:24.088253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:41037 (local address 127.9.12.193:49753)
0928 15:02:24.089016 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:24.089058 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:24.090961 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:24.091698 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:24.091707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:24.092315 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:24.092931 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:24.092968 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.094017 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.094022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:24.095238 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:24.095250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.096074 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.096081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:24.096378 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:24.097054 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:24.097083 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:24.099238 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:24.103932 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:24.103939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:24.103999 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:24.104682 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:24.105109 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:24.105113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:24.105115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:24.105314 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:24.105856 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:24.105864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:24.106224 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 15:02:24.106501 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 15:02:24.107079 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":286,"threads_started":1}
I20260928 15:02:24.109923 9508 heartbeater.cc:347] Connected to a master server at 127.9.12.254:41037
I20260928 15:02:24.110241 9508 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:24.111037 9508 heartbeater.cc:511] Master 127.9.12.254:41037 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:02:24.113303 9319 ts_manager.cc:194] Registered new tserver with Master: b61078b96bd24f6cb4c080ff4c009f29 (127.9.12.193:39637)
I20260928 15:02:24.114813 9319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:49753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:24.132929 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41037
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:24.474272 9514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:24.474591 9514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:24.474704 9514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:24.482249 9514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:24.482391 9514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:24.482550 9514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:24.492190 9514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:24.493487 9514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:24.494756 9514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:24.508204 9519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:24.508145 9520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:24.508680 9522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:24.508971 9514 server_base.cc:1034] running on GCE node
I20260928 15:02:24.509541 9514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:24.510349 9514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:24.511572 9514 hybrid_clock.cc:648] HybridClock initialized: now 1790607744511528 us; error 59 us; skew 500 ppm
I20260928 15:02:24.511878 9514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:24 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607744, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:24.514859 9514 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:24.516350 9514 webserver.cc:460] Webserver started at http://127.9.12.194:39213/ using document root <none> and password file <none>
I20260928 15:02:24.517014 9514 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:24.517172 9514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:24.517509 9514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:24.520659 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "39f69ec2cf4543319fd70b3574ef58dd"
format_stamp: "Formatted at 2026-09-28 15:02:24 on dist-test-slave-dmhs"
server_key: "f6edfe1918de38e9770df91f858be91a"
server_key_iv: "2bc198ee96a16115cbcc266ff8617633"
server_key_version: "encryptionkey@0"
I20260928 15:02:24.521452 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "39f69ec2cf4543319fd70b3574ef58dd"
format_stamp: "Formatted at 2026-09-28 15:02:24 on dist-test-slave-dmhs"
server_key: "f6edfe1918de38e9770df91f858be91a"
server_key_iv: "2bc198ee96a16115cbcc266ff8617633"
server_key_version: "encryptionkey@0"
I20260928 15:02:24.526528 9514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:02:24.530040 9529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:24.531282 9514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:02:24.531451 9514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "39f69ec2cf4543319fd70b3574ef58dd"
format_stamp: "Formatted at 2026-09-28 15:02:24 on dist-test-slave-dmhs"
server_key: "f6edfe1918de38e9770df91f858be91a"
server_key_iv: "2bc198ee96a16115cbcc266ff8617633"
server_key_version: "encryptionkey@0"
I20260928 15:02:24.531665 9514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:24.558589 9514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:24.563246 9514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:24.563549 9514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:24.564503 9514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:24.565752 9514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:24.565853 9514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:24.566000 9514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:24.566080 9514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:24.603160 9514 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:40531
I20260928 15:02:24.603339 9641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:40531 every 8 connection(s)
I20260928 15:02:24.604553 9514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:24.609230 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 9514
I20260928 15:02:24.609427 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:24.609849 9267 external_mini_cluster.cc:1428] Setting key dcc7d43332f412c35d27d335afa1c330
Sep 28 15:02:24 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607744, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:24.626276 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:24.607267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:38455 (local address 127.9.12.254:41037)
0928 15:02:24.607451 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:24.607456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:24.608571 (+ 1115us) server_negotiation.cc:407] Connection header received
0928 15:02:24.610792 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:24.610799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:24.610891 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:24.611001 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:24.612528 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.613882 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.615439 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.616490 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.616608 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:24.620530 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:24.620535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:24.620538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:24.620571 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:24.623272 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:24.623986 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:24.623991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:24.623994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:24.624122 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:24.624589 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:24.624594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:24.624597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:24.624889 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:24.625060 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:24.625711 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 15:02:24.625944 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:02:24.627457 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:24.607681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:41037 (local address 127.9.12.194:38455)
0928 15:02:24.608412 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:24.608461 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:24.610464 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:24.611187 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:24.611195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:24.611772 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:24.612364 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:24.612383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.614048 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.614052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:24.615298 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:24.615310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:24.616732 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:24.616741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:24.617069 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:24.617801 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:24.617832 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:24.620307 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:24.623411 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:24.623429 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:24.623445 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:24.623857 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:24.624239 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:24.624244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:24.624248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:24.624470 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:24.625026 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:24.625034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:24.625357 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 15:02:24.625623 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 15:02:24.626121 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260928 15:02:24.628947 9642 heartbeater.cc:347] Connected to a master server at 127.9.12.254:41037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
I20260928 15:02:24.629287 9642 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:24.629968 9642 heartbeater.cc:511] Master 127.9.12.254:41037 requested a full tablet report, sending...
I20260928 15:02:24.631363 9319 ts_manager.cc:194] Registered new tserver with Master: 39f69ec2cf4543319fd70b3574ef58dd (127.9.12.194:40531)
I20260928 15:02:24.632144 9319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:38455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:24.649596 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41037
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:24.994760 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:24.995100 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:24.995311 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:25.002661 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:25.002832 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:25.003014 9647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:25.013216 9647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:25.014465 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:25.015796 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:25.030174 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:25.030236 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:25.030134 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:25.031064 9647 server_base.cc:1034] running on GCE node
I20260928 15:02:25.031647 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:25.032794 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:25.034004 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790607745033947 us; error 49 us; skew 500 ppm
I20260928 15:02:25.034314 9647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:25.037936 9647 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:25.039276 9647 webserver.cc:460] Webserver started at http://127.9.12.195:33655/ using document root <none> and password file <none>
I20260928 15:02:25.039886 9647 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:25.039984 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:25.040238 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:25.043020 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "62da10597d534be99b33b9f4d269adef"
format_stamp: "Formatted at 2026-09-28 15:02:25 on dist-test-slave-dmhs"
server_key: "359c419957dc7024b61e82fddcbf0ceb"
server_key_iv: "af5fb9dc4562673b5defcbad8586ca38"
server_key_version: "encryptionkey@0"
I20260928 15:02:25.043731 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "62da10597d534be99b33b9f4d269adef"
format_stamp: "Formatted at 2026-09-28 15:02:25 on dist-test-slave-dmhs"
server_key: "359c419957dc7024b61e82fddcbf0ceb"
server_key_iv: "af5fb9dc4562673b5defcbad8586ca38"
server_key_version: "encryptionkey@0"
I20260928 15:02:25.048768 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:02:25.052011 9662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:25.053325 9647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:25.053448 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "62da10597d534be99b33b9f4d269adef"
format_stamp: "Formatted at 2026-09-28 15:02:25 on dist-test-slave-dmhs"
server_key: "359c419957dc7024b61e82fddcbf0ceb"
server_key_iv: "af5fb9dc4562673b5defcbad8586ca38"
server_key_version: "encryptionkey@0"
I20260928 15:02:25.053607 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:25.066349 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:25.070757 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:25.070986 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:25.071827 9647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:25.073328 9647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:25.073401 9647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:25.073482 9647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:25.073532 9647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:25.112324 9647 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:38225
I20260928 15:02:25.112424 9774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:38225 every 8 connection(s)
I20260928 15:02:25.113659 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:25.115078 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 9647
I20260928 15:02:25.115315 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:25.115777 9267 external_mini_cluster.cc:1428] Setting key 1fb66bb37df65a0e9c34a8d7f69526c1
I20260928 15:02:25.118116 9508 heartbeater.cc:503] Master 127.9.12.254:41037 was elected leader, sending a full tablet report...
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:25.134649 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.116315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:46251 (local address 127.9.12.254:41037)
0928 15:02:25.116530 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:25.116536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:25.118627 (+ 2091us) server_negotiation.cc:407] Connection header received
0928 15:02:25.121090 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:25.121094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:25.121158 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:25.121237 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:25.122746 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.123624 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.125218 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.125603 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.125694 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.129030 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:25.129034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:25.129037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:25.129067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:25.131852 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.132696 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.132701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.132704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.132857 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.133235 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.133238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.133240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.133496 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:25.133620 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:25.134240 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 15:02:25.134391 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:02:25.135753 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.116980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:41037 (local address 127.9.12.195:46251)
0928 15:02:25.118467 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:25.118521 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:25.120819 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:25.121434 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:25.121441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:25.122013 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:25.122598 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:25.122615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.123782 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.123787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:25.125096 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:25.125108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.125783 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.125788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:25.125975 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.126657 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:25.126678 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:25.128848 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:25.132013 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:25.132029 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:25.132042 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:25.132553 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:25.132950 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:25.132953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:25.132955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:25.133144 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:25.133621 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:25.133627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:25.133938 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 15:02:25.134154 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 15:02:25.134615 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":734,"threads_started":1}
I20260928 15:02:25.137115 9775 heartbeater.cc:347] Connected to a master server at 127.9.12.254:41037
I20260928 15:02:25.137439 9775 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:25.138151 9775 heartbeater.cc:511] Master 127.9.12.254:41037 requested a full tablet report, sending...
I20260928 15:02:25.139438 9318 ts_manager.cc:194] Registered new tserver with Master: 62da10597d534be99b33b9f4d269adef (127.9.12.195:38225)
I20260928 15:02:25.140611 9318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:46251
I20260928 15:02:25.144855 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:25.188021 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.172566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.9.12.193:39637)
0928 15:02:25.173438 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:25.173445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:25.173469 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:25.173529 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:25.173535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:25.173736 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:25.173924 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:25.175676 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.176748 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.178022 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.178569 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.178708 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.182610 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:25.182621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:25.182633 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:25.182664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:25.185527 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.186119 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.186125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.186127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.186252 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.186581 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.186584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.186586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.186916 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:25.187166 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:25.187378 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:02:25.187558 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":510,"threads_started":1}
W20260928 15:02:25.189309 9423 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50046
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:25.232466 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.218904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.9.12.193:39637)
0928 15:02:25.219115 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:25.219121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:25.219225 (+ 104us) server_negotiation.cc:407] Connection header received
0928 15:02:25.219359 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:25.219363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:25.219452 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:25.219537 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:25.220894 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.221566 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.222932 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.223460 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.223595 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.227494 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:25.227499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:25.227503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:25.227535 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:25.229992 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.230631 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.230637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.230642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.230763 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.231123 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.231129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.231132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.231393 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:25.231625 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:25.231920 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:02:25.232140 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:02:25.233942 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 9380
I20260928 15:02:25.245743 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.234821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.9.12.193:39637)
0928 15:02:25.234985 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:25.234988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:25.235367 (+ 379us) server_negotiation.cc:407] Connection header received
0928 15:02:25.235499 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:25.235502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:25.235551 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:25.235667 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:25.236811 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.237455 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.238618 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.239011 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.239100 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.241194 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:25.241198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:25.241201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:25.241228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:25.243657 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.244219 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.244222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.244224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.244406 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.244689 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.244692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.244694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.244916 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:25.245083 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:25.245330 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:02:25.245469 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:02:25.246428 9503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:25.327348 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 9380
I20260928 15:02:25.343237 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 9514
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:25.358471 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.344110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.9.12.194:40531)
0928 15:02:25.344636 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:25.344643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:25.344667 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:25.344725 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:25.344731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:25.344934 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:25.345050 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:25.346180 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.347205 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.348453 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.348963 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.349059 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.353287 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:25.353297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:25.353310 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:25.353339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:25.355893 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.356573 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.356580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.356582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.356701 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.357101 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.357106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.357109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.357436 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:25.357670 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:25.357915 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 15:02:25.358102 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":295,"threads_started":1}
I20260928 15:02:25.359691 9637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:25.440706 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 9514
I20260928 15:02:25.456048 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 9647
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:25.471765 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:25.457205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.9.12.195:38225)
0928 15:02:25.457889 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:25.457895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:25.457922 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:25.457980 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:25.457985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:25.458186 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:25.458351 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:25.459592 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.460700 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.461931 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:25.462445 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:25.462585 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:25.466561 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:25.466577 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:25.466590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:25.466619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:25.469090 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.469726 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.469751 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.469754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.469867 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:25.470235 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:25.470241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:25.470244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:25.470563 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:25.470893 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:25.471190 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 15:02:25.471444 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":407,"threads_started":1}
I20260928 15:02:25.472981 9770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:25.553330 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 9647
I20260928 15:02:25.568886 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 9287
I20260928 15:02:25.570091 9349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:25.674173 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 9287
2026-09-28T15:02:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3906 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9796](info): set up 2 sockets
Sep 28 15:02:25 dist-test-slave-dmhs krb5kdc[9796](info): commencing operation
krb5kdc: starting...
W20260928 15:02:26.744516 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:26 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:26.919672 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:35137
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:27.236521 9809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:27.236802 9809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:27.236893 9809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:27.243973 9809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:27.244087 9809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:27.244132 9809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:27.244190 9809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:27.244233 9809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:27.253934 9809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:35137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:27.255226 9809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:27.256568 9809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:27.268055 9814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:27.267998 9817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:27.267970 9815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:27.268895 9809 server_base.cc:1034] running on GCE node
I20260928 15:02:27.269577 9809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:27.270844 9809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:27.272033 9809 hybrid_clock.cc:648] HybridClock initialized: now 1790607747271981 us; error 48 us; skew 500 ppm
I20260928 15:02:27.272421 9809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:27 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:27.275470 9809 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:27.277115 9809 webserver.cc:460] Webserver started at http://127.9.12.254:43235/ using document root <none> and password file <none>
I20260928 15:02:27.277794 9809 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:27.277937 9809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:27.278234 9809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:27.281129 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "35e24d6a589a47d4a113b012be3e73c1"
format_stamp: "Formatted at 2026-09-28 15:02:27 on dist-test-slave-dmhs"
server_key: "29144e6e50518d4d614660dad6b4b64e"
server_key_iv: "1f41087bd884a915695dfb9f4a7db950"
server_key_version: "encryptionkey@0"
I20260928 15:02:27.281859 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "35e24d6a589a47d4a113b012be3e73c1"
format_stamp: "Formatted at 2026-09-28 15:02:27 on dist-test-slave-dmhs"
server_key: "29144e6e50518d4d614660dad6b4b64e"
server_key_iv: "1f41087bd884a915695dfb9f4a7db950"
server_key_version: "encryptionkey@0"
I20260928 15:02:27.286684 9809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:02:27.289897 9824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:27.291064 9809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:27.291245 9809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "35e24d6a589a47d4a113b012be3e73c1"
format_stamp: "Formatted at 2026-09-28 15:02:27 on dist-test-slave-dmhs"
server_key: "29144e6e50518d4d614660dad6b4b64e"
server_key_iv: "1f41087bd884a915695dfb9f4a7db950"
server_key_version: "encryptionkey@0"
I20260928 15:02:27.291426 9809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:27.309634 9809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:27.314128 9809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:27.314455 9809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:27.332196 9809 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:35137
I20260928 15:02:27.332237 9875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:35137 every 8 connection(s)
I20260928 15:02:27.333549 9809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:27.335702 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 9809
I20260928 15:02:27.335933 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:27.336663 9267 external_mini_cluster.cc:1428] Setting key 033e64447a7ba7674b6c4af0fc9e9c64
I20260928 15:02:27.337304 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:02:27 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:27.351812 9876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1: Bootstrap starting.
I20260928 15:02:27.355873 9876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:27.357224 9876 log.cc:824] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:27.359578 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:27.338496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.9.12.254:35137)
0928 15:02:27.339215 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:27.339225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:27.339262 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:02:27.340795 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:27.340815 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:27.341188 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:27.341493 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:27.343281 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:27.344641 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:27.346368 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:27.346998 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:27.347129 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:27.351630 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:27.351641 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:27.351661 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:27.351708 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:27.355464 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:27.356142 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:27.356150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:27.356157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:27.356424 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:27.356768 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:27.356773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:27.356776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:27.357463 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:27.357786 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:27.358240 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 15:02:27.358601 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":348,"threads_started":1}
I20260928 15:02:27.360193 9876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1: No bootstrap required, opened a new log
I20260928 15:02:27.367847 9876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e24d6a589a47d4a113b012be3e73c1" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35137 } }
I20260928 15:02:27.368211 9876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:27.368348 9876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e24d6a589a47d4a113b012be3e73c1, State: Initialized, Role: FOLLOWER
I20260928 15:02:27.368845 9876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e24d6a589a47d4a113b012be3e73c1" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35137 } }
I20260928 15:02:27.369060 9876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:27.369194 9876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:27.369346 9876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:27.371382 9876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e24d6a589a47d4a113b012be3e73c1" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35137 } }
I20260928 15:02:27.371835 9876 leader_election.cc:302] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e24d6a589a47d4a113b012be3e73c1; no voters:
I20260928 15:02:27.372428 9876 leader_election.cc:288] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:27.372551 9881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:27.373595 9881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [term 1 LEADER]: Becoming Leader. State: Replica: 35e24d6a589a47d4a113b012be3e73c1, State: Running, Role: LEADER
I20260928 15:02:27.374156 9881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e24d6a589a47d4a113b012be3e73c1" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35137 } }
I20260928 15:02:27.374627 9876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:27.377053 9882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35e24d6a589a47d4a113b012be3e73c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e24d6a589a47d4a113b012be3e73c1" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35137 } } }
I20260928 15:02:27.377436 9882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:27.377022 9883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e24d6a589a47d4a113b012be3e73c1. Latest consensus state: current_term: 1 leader_uuid: "35e24d6a589a47d4a113b012be3e73c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e24d6a589a47d4a113b012be3e73c1" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35137 } } }
I20260928 15:02:27.377803 9883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:27.380348 9887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:27.384752 9887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:27.392927 9887 catalog_manager.cc:1347] Generated new cluster ID: 34ce998fccd844e6913cea6e793040ce
I20260928 15:02:27.393059 9887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:27.409626 9887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:27.410859 9887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:27.420450 9887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35e24d6a589a47d4a113b012be3e73c1: Generated new TSK 0
I20260928 15:02:27.421221 9887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:27.464212 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:35137
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:27.787781 9902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:27.788079 9902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:27.788249 9902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:27.795907 9902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:27.796054 9902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:27.796235 9902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:27.806154 9902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:35137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:27.807525 9902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:27.809000 9902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:27.822718 9907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:27.822664 9908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:27.822743 9910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:27.823740 9902 server_base.cc:1034] running on GCE node
I20260928 15:02:27.824434 9902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:27.825399 9902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:27.826629 9902 hybrid_clock.cc:648] HybridClock initialized: now 1790607747826577 us; error 47 us; skew 500 ppm
I20260928 15:02:27.826973 9902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:27 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:27.830111 9902 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:27.831655 9902 webserver.cc:460] Webserver started at http://127.9.12.193:33301/ using document root <none> and password file <none>
I20260928 15:02:27.832435 9902 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:27.832576 9902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:27.832886 9902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:27.835762 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "dee122209eeb4c2281feaeb5eb5812bb"
format_stamp: "Formatted at 2026-09-28 15:02:27 on dist-test-slave-dmhs"
server_key: "87f95b0894f8d7d54307f7302fc584cd"
server_key_iv: "a99b0f0f7583dd7954ab19756f1776f1"
server_key_version: "encryptionkey@0"
I20260928 15:02:27.836603 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "dee122209eeb4c2281feaeb5eb5812bb"
format_stamp: "Formatted at 2026-09-28 15:02:27 on dist-test-slave-dmhs"
server_key: "87f95b0894f8d7d54307f7302fc584cd"
server_key_iv: "a99b0f0f7583dd7954ab19756f1776f1"
server_key_version: "encryptionkey@0"
I20260928 15:02:27.841686 9902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:02:27.845136 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:27.846421 9902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:02:27.846604 9902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "dee122209eeb4c2281feaeb5eb5812bb"
format_stamp: "Formatted at 2026-09-28 15:02:27 on dist-test-slave-dmhs"
server_key: "87f95b0894f8d7d54307f7302fc584cd"
server_key_iv: "a99b0f0f7583dd7954ab19756f1776f1"
server_key_version: "encryptionkey@0"
I20260928 15:02:27.846838 9902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:27.887092 9902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:27.891177 9902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:27.891492 9902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:27.892597 9902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:27.893939 9902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:27.894039 9902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:27.894181 9902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:27.894255 9902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:27.927752 9902 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:42795
I20260928 15:02:27.927839 10029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:42795 every 8 connection(s)
I20260928 15:02:27.929133 9902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:27.933399 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 9902
I20260928 15:02:27.933625 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:27.934430 9267 external_mini_cluster.cc:1428] Setting key add37122bed2fdff692ddd1a05efaee7
Sep 28 15:02:27 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:27.952638 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:27.931619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:57687 (local address 127.9.12.254:35137)
0928 15:02:27.932066 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:27.932070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:27.933004 (+ 934us) server_negotiation.cc:407] Connection header received
0928 15:02:27.935499 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:27.935506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:27.935604 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:27.935732 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:27.937724 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:27.938413 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:27.940967 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:27.941384 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:27.941498 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:27.945135 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:27.945145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:27.945152 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:27.945198 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:27.948907 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:27.949881 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:27.949885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:27.949887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:27.950007 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:27.950585 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:27.950591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:27.950596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:27.950873 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:27.951024 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:27.951903 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 15:02:27.952159 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":239,"threads_started":1}
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:02:27.953415 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:27.932102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:35137 (local address 127.9.12.193:57687)
0928 15:02:27.932818 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:27.932861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:27.935231 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:27.935984 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:27.935992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:27.936697 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:27.937513 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:27.937556 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:27.938587 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:27.938595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:27.940828 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:27.940846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:27.941631 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:27.941638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:27.941942 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:27.942716 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:27.942741 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:27.944881 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:27.949052 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:27.949058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:27.949072 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:27.949755 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:27.950122 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:27.950128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:27.950135 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:27.950443 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:27.951008 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:27.951016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:27.951452 (+ 436us) client_negotiation.cc:769] Sending connection context
0928 15:02:27.951743 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 15:02:27.952150 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":322,"threads_started":1}
I20260928 15:02:27.954969 10030 heartbeater.cc:347] Connected to a master server at 127.9.12.254:35137
I20260928 15:02:27.955293 10030 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:02:27.956132 10030 heartbeater.cc:511] Master 127.9.12.254:35137 requested a full tablet report, sending...
I20260928 15:02:27.958256 9841 ts_manager.cc:194] Registered new tserver with Master: dee122209eeb4c2281feaeb5eb5812bb (127.9.12.193:42795)
I20260928 15:02:27.959662 9841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:57687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:27.977329 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:35137
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:28.289353 10036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:28.289635 10036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:28.289736 10036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:28.297020 10036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:28.297127 10036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:28.297243 10036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:28.306653 10036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:35137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:28.307878 10036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:28.309231 10036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:28.321971 10041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:28.322021 10043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:28.321985 10045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:28.323127 10036 server_base.cc:1034] running on GCE node
I20260928 15:02:28.323668 10036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:28.324587 10036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:28.325785 10036 hybrid_clock.cc:648] HybridClock initialized: now 1790607748325745 us; error 29 us; skew 500 ppm
I20260928 15:02:28.326056 10036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:28 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:28.329154 10036 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:28.330658 10036 webserver.cc:460] Webserver started at http://127.9.12.194:40253/ using document root <none> and password file <none>
I20260928 15:02:28.331341 10036 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:28.331439 10036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:28.331743 10036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:28.334600 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "5f623016996240e4b5f8031daa49cca8"
format_stamp: "Formatted at 2026-09-28 15:02:28 on dist-test-slave-dmhs"
server_key: "9bdbbdce28ebe098198b82b1813967c9"
server_key_iv: "bddedff526b4c8daf5813459c7db3a92"
server_key_version: "encryptionkey@0"
I20260928 15:02:28.335296 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "5f623016996240e4b5f8031daa49cca8"
format_stamp: "Formatted at 2026-09-28 15:02:28 on dist-test-slave-dmhs"
server_key: "9bdbbdce28ebe098198b82b1813967c9"
server_key_iv: "bddedff526b4c8daf5813459c7db3a92"
server_key_version: "encryptionkey@0"
I20260928 15:02:28.340374 10036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 15:02:28.343786 10052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:28.345222 10036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:28.345386 10036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "5f623016996240e4b5f8031daa49cca8"
format_stamp: "Formatted at 2026-09-28 15:02:28 on dist-test-slave-dmhs"
server_key: "9bdbbdce28ebe098198b82b1813967c9"
server_key_iv: "bddedff526b4c8daf5813459c7db3a92"
server_key_version: "encryptionkey@0"
I20260928 15:02:28.345585 10036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:28.377450 10036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:28.381651 10036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:28.381953 10036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:28.382980 10036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:28.384300 10036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:28.384407 10036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:28.384563 10036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:28.384637 10036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:28.417135 10036 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:38223
I20260928 15:02:28.417179 10164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:38223 every 8 connection(s)
I20260928 15:02:28.418434 10036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:28.423729 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10036
I20260928 15:02:28.423920 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:28.424455 9267 external_mini_cluster.cc:1428] Setting key b1f197e402c1cab233a1a89bab134de3
Sep 28 15:02:28 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:28.441279 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.420976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:40035 (local address 127.9.12.254:35137)
0928 15:02:28.421194 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:28.421201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:28.422400 (+ 1199us) server_negotiation.cc:407] Connection header received
0928 15:02:28.424681 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:28.424685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:28.424756 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:28.424843 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:28.426644 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.427361 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.429952 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.430375 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.430471 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:28.434181 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:28.434190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:28.434198 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:28.434244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:28.437911 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:28.438638 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:28.438646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:28.438652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:28.438813 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:28.439264 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:28.439269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:28.439272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:28.439690 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:28.439856 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:28.440609 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 15:02:28.440871 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:02:28.442015 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.421554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:35137 (local address 127.9.12.194:40035)
0928 15:02:28.422240 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:28.422285 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:28.424341 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:28.425146 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:28.425164 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:28.425741 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:28.426487 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:28.426506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.427525 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.427534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:28.429800 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:28.429819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.430601 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.430611 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:28.430960 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:28.431721 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:28.431754 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:28.433947 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:28.438063 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:28.438075 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:28.438089 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:28.438497 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:28.438932 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:28.438936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:28.438938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:28.439133 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:28.439823 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:28.439829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:28.440165 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 15:02:28.440453 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 15:02:28.440849 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:02:28.443569 10165 heartbeater.cc:347] Connected to a master server at 127.9.12.254:35137
I20260928 15:02:28.443881 10165 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:28.444680 10165 heartbeater.cc:511] Master 127.9.12.254:35137 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
I20260928 15:02:28.446444 9841 ts_manager.cc:194] Registered new tserver with Master: 5f623016996240e4b5f8031daa49cca8 (127.9.12.194:38223)
I20260928 15:02:28.447759 9841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:40035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:28.469333 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:35137
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:28.798980 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:28.799297 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:28.799476 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:28.808890 10170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:28.809038 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:28.809228 10170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:28.819108 10170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:35137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:28.820417 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:28.821745 10170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:28.834625 10176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:28.834699 10175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:28.834743 10178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:28.835597 10170 server_base.cc:1034] running on GCE node
I20260928 15:02:28.836251 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:28.837174 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:28.838364 10170 hybrid_clock.cc:648] HybridClock initialized: now 1790607748838315 us; error 55 us; skew 500 ppm
I20260928 15:02:28.838663 10170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:28 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:28.846954 10170 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:28.848675 10170 webserver.cc:460] Webserver started at http://127.9.12.195:36127/ using document root <none> and password file <none>
I20260928 15:02:28.849370 10170 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:28.849501 10170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:28.849804 10170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:28.852754 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "a5bf3506b59440b98d80dc408eb292e1"
format_stamp: "Formatted at 2026-09-28 15:02:28 on dist-test-slave-dmhs"
server_key: "ea8ddd023af54e237775ba9834d88ca5"
server_key_iv: "0a93392f7ff1dc9131ae6198e0441cbd"
server_key_version: "encryptionkey@0"
I20260928 15:02:28.853497 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "a5bf3506b59440b98d80dc408eb292e1"
format_stamp: "Formatted at 2026-09-28 15:02:28 on dist-test-slave-dmhs"
server_key: "ea8ddd023af54e237775ba9834d88ca5"
server_key_iv: "0a93392f7ff1dc9131ae6198e0441cbd"
server_key_version: "encryptionkey@0"
I20260928 15:02:28.858711 10170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:02:28.862033 10185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:28.863201 10170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:02:28.863363 10170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "a5bf3506b59440b98d80dc408eb292e1"
format_stamp: "Formatted at 2026-09-28 15:02:28 on dist-test-slave-dmhs"
server_key: "ea8ddd023af54e237775ba9834d88ca5"
server_key_iv: "0a93392f7ff1dc9131ae6198e0441cbd"
server_key_version: "encryptionkey@0"
I20260928 15:02:28.863559 10170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:28.877522 10170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:28.881469 10170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:28.881719 10170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:28.882751 10170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:28.884001 10170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:28.884109 10170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:28.884296 10170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:28.884369 10170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 15:02:28.923465 10170 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:46609
I20260928 15:02:28.923512 10297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:46609 every 8 connection(s)
I20260928 15:02:28.924880 10170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:28.928197 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10170
I20260928 15:02:28.928416 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:28.929003 9267 external_mini_cluster.cc:1428] Setting key c0a7f72810df64095d5f90b21ef2a68f
Sep 28 15:02:28 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:28.945472 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.927706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:46671 (local address 127.9.12.254:35137)
0928 15:02:28.927925 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:28.927932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:28.929422 (+ 1490us) server_negotiation.cc:407] Connection header received
0928 15:02:28.931762 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:28.931767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:28.931861 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:28.931973 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:28.933567 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.934392 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.936110 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.936624 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.936763 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:28.940122 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:28.940129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:28.940133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:28.940175 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:28.942560 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:28.943306 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:28.943313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:28.943316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:28.943480 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:28.943929 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:28.943933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:28.943935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:28.944256 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:28.944424 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:28.945071 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 15:02:28.945220 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:02:28.946467 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.928314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:35137 (local address 127.9.12.195:46671)
0928 15:02:28.929247 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:28.929296 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:28.931508 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:28.932370 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:28.932381 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:28.932859 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:28.933407 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:28.933424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.934623 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.934629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:28.935952 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:28.935968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.936861 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.936871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:28.937130 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:28.937810 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:28.937832 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:28.939917 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:28.942699 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:28.942709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:28.942726 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:28.943198 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:28.943586 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:28.943589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:28.943591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:28.943834 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:28.944419 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:28.944426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:28.944776 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 15:02:28.944989 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 15:02:28.945327 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":359,"threads_started":1}
I20260928 15:02:28.947793 10298 heartbeater.cc:347] Connected to a master server at 127.9.12.254:35137
I20260928 15:02:28.948083 10298 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:28.948909 10298 heartbeater.cc:511] Master 127.9.12.254:35137 requested a full tablet report, sending...
I20260928 15:02:28.950404 9841 ts_manager.cc:194] Registered new tserver with Master: a5bf3506b59440b98d80dc408eb292e1 (127.9.12.195:46609)
I20260928 15:02:28.951169 9841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:46671
I20260928 15:02:28.956244 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:28.962460 10030 heartbeater.cc:503] Master 127.9.12.254:35137 was elected leader, sending a full tablet report...
I20260928 15:02:28.974220 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.962436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.9.12.254:35137)
0928 15:02:28.962628 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:28.962637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:28.963191 (+ 554us) server_negotiation.cc:407] Connection header received
0928 15:02:28.963320 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:28.963325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:28.963392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:28.963491 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:28.965230 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.965898 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.967103 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:28.967473 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:28.967574 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:28.969752 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:28.969757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:28.969759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:28.969786 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:28.972013 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:28.972616 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:28.972619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:28.972621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:28.972745 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:28.973042 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:28.973045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:28.973047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:28.973325 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:28.973496 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:28.973749 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:02:28.973892 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:02:28.980342 9841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:28.987218 9841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:29.009891 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.999435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.9.12.195:46609)
0928 15:02:29.000452 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.000464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.002197 (+ 1733us) server_negotiation.cc:407] Connection header received
0928 15:02:29.002505 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.002514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.002791 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.002969 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:29.003696 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.005340 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.007511 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.008783 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.008905 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.009132 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.009393 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.009547 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":649,"threads_started":1}
I20260928 15:02:29.010004 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.999451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:46609 (local address 127.0.0.1:50564)
0928 15:02:29.002063 (+ 2612us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:29.002103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:29.002275 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:29.003008 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:29.003015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:29.003056 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:29.003450 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.003464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.005496 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.005501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.007323 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.007341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.008977 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.008984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.009154 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.009210 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:02:29.009341 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:02:29.009523 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2456,"mutex_wait_us":96,"thread_start_us":2846,"threads_started":1}
I20260928 15:02:29.010743 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:28.999451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:38223 (local address 127.0.0.1:33950)
0928 15:02:29.002063 (+ 2612us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:29.002103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:29.002268 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:29.003411 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:29.003415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:29.003439 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:29.003880 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.003894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.005467 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.005475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.007707 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.007722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.009908 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.009916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.010122 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.010156 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:02:29.010246 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:02:29.010374 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2339,"mutex_wait_us":1,"thread_start_us":2246,"threads_started":1}
I20260928 15:02:29.010952 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.000203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.9.12.194:38223)
0928 15:02:29.002862 (+ 2659us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.002869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.002896 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:29.002990 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.002998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.003216 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.003411 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:29.004093 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.005331 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.008131 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.009446 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.009563 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.009765 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.010444 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.010628 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2519,"thread_start_us":2550,"threads_started":1}
I20260928 15:02:29.011242 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.001661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:42795 (local address 127.0.0.1:43682)
0928 15:02:29.002885 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:29.002902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:29.003103 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:29.004044 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:29.004049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:29.004073 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:29.004476 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.004488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.006286 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.006291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.008396 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.008411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.010441 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.010448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.010643 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.010675 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:02:29.010761 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:02:29.010870 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"spinlock_wait_cycles":6656,"thread_start_us":959,"threads_started":1}
I20260928 15:02:29.011799 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.000747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.9.12.193:42795)
0928 15:02:29.001540 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.001546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.002982 (+ 1436us) server_negotiation.cc:407] Connection header received
0928 15:02:29.003321 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.003327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.003554 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.003754 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:29.004662 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.006137 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.008565 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.009924 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.010083 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.010292 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.011286 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.011449 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":579,"threads_started":1}
I20260928 15:02:29.014248 10233 tablet_service.cc:1467] Processing CreateTablet for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8 (DEFAULT_TABLE table=test-table [id=ed94a7aeec884354b55d2eb4f2606a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:29.014451 10100 tablet_service.cc:1467] Processing CreateTablet for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8 (DEFAULT_TABLE table=test-table [id=ed94a7aeec884354b55d2eb4f2606a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:29.015784 10100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:29.015796 10233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:29.015475 9965 tablet_service.cc:1467] Processing CreateTablet for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8 (DEFAULT_TABLE table=test-table [id=ed94a7aeec884354b55d2eb4f2606a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:29.016736 9965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:29.026865 10316 tablet_bootstrap.cc:492] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: Bootstrap starting.
I20260928 15:02:29.031407 10316 tablet_bootstrap.cc:654] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:29.032112 10317 tablet_bootstrap.cc:492] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: Bootstrap starting.
I20260928 15:02:29.033372 10316 log.cc:824] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:29.034734 10318 tablet_bootstrap.cc:492] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: Bootstrap starting.
I20260928 15:02:29.036228 10317 tablet_bootstrap.cc:654] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:29.036460 10316 tablet_bootstrap.cc:492] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: No bootstrap required, opened a new log
I20260928 15:02:29.036729 10316 ts_tablet_manager.cc:1397] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 15:02:29.037550 10317 log.cc:824] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:29.038288 10318 tablet_bootstrap.cc:654] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:29.039713 10318 log.cc:824] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:29.042852 10317 tablet_bootstrap.cc:492] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: No bootstrap required, opened a new log
I20260928 15:02:29.043080 10318 tablet_bootstrap.cc:492] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: No bootstrap required, opened a new log
I20260928 15:02:29.043149 10317 ts_tablet_manager.cc:1397] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260928 15:02:29.043350 10318 ts_tablet_manager.cc:1397] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 15:02:29.048532 10316 raft_consensus.cc:348] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.049088 10316 raft_consensus.cc:374] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:29.049192 10316 raft_consensus.cc:729] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f623016996240e4b5f8031daa49cca8, State: Initialized, Role: FOLLOWER
I20260928 15:02:29.049857 10316 consensus_queue.cc:260] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.050716 10317 raft_consensus.cc:348] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.051450 10165 heartbeater.cc:503] Master 127.9.12.254:35137 was elected leader, sending a full tablet report...
I20260928 15:02:29.052351 10316 ts_tablet_manager.cc:1428] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260928 15:02:29.052597 10317 raft_consensus.cc:374] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:29.052722 10317 raft_consensus.cc:729] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bf3506b59440b98d80dc408eb292e1, State: Initialized, Role: FOLLOWER
I20260928 15:02:29.053280 10317 consensus_queue.cc:260] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.054441 10298 heartbeater.cc:503] Master 127.9.12.254:35137 was elected leader, sending a full tablet report...
I20260928 15:02:29.055043 10317 ts_tablet_manager.cc:1428] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260928 15:02:29.055383 10318 raft_consensus.cc:348] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.055842 10318 raft_consensus.cc:374] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:29.055979 10318 raft_consensus.cc:729] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee122209eeb4c2281feaeb5eb5812bb, State: Initialized, Role: FOLLOWER
I20260928 15:02:29.056627 10318 consensus_queue.cc:260] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.058241 10318 ts_tablet_manager.cc:1428] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260928 15:02:29.170535 10166 tablet.cc:2369] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:29.176507 10299 tablet.cc:2369] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:29.181342 10031 tablet.cc:2369] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:29.473134 10324 raft_consensus.cc:482] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:29.473431 10324 raft_consensus.cc:504] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.475410 10324 leader_election.cc:288] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5bf3506b59440b98d80dc408eb292e1 (127.9.12.195:46609), 5f623016996240e4b5f8031daa49cca8 (127.9.12.194:38223)
I20260928 15:02:29.485144 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.476519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:33369 (local address 127.9.12.194:38223)
0928 15:02:29.476745 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.476751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.477501 (+ 750us) server_negotiation.cc:407] Connection header received
0928 15:02:29.477671 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.477675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.477732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.477803 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:29.478325 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.479427 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.482559 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.483936 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.484135 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.484212 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.484736 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.484873 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:02:29.485144 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.475663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:39435 (local address 127.9.12.195:46609)
0928 15:02:29.475892 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.475902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.476900 (+ 998us) server_negotiation.cc:407] Connection header received
0928 15:02:29.477066 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.477075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.477166 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.477304 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:29.478019 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.479365 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.482583 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.484030 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.484169 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.484234 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.484736 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.484873 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:02:29.485409 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.475786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:38223 (local address 127.9.12.193:33369)
0928 15:02:29.477379 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:29.477402 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:29.477547 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:29.477841 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:29.477848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:29.477875 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:29.478188 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.478195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.479531 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.479536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.482258 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.482273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.484173 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.484182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.484431 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.484598 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 15:02:29.484694 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 15:02:29.484933 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"mutex_wait_us":256,"thread_start_us":1261,"threads_started":1}
I20260928 15:02:29.485719 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.475601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:46609 (local address 127.9.12.193:39435)
0928 15:02:29.476767 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:29.476786 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:29.476918 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:29.477340 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:29.477345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:29.477378 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:29.477763 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.477772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.479506 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.479513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.482258 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:29.482273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.484213 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.484218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:29.484431 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.484598 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 15:02:29.484694 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 15:02:29.484933 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":323,"thread_start_us":716,"threads_started":1}
I20260928 15:02:29.486616 10253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65d4c5fa5c64532a2f8a1b8ad70cda8" candidate_uuid: "dee122209eeb4c2281feaeb5eb5812bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bf3506b59440b98d80dc408eb292e1" is_pre_election: true
I20260928 15:02:29.486675 10120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65d4c5fa5c64532a2f8a1b8ad70cda8" candidate_uuid: "dee122209eeb4c2281feaeb5eb5812bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f623016996240e4b5f8031daa49cca8" is_pre_election: true
I20260928 15:02:29.487365 10253 raft_consensus.cc:2454] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee122209eeb4c2281feaeb5eb5812bb in term 0.
I20260928 15:02:29.487479 10120 raft_consensus.cc:2454] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee122209eeb4c2281feaeb5eb5812bb in term 0.
I20260928 15:02:29.488154 9919 leader_election.cc:302] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5bf3506b59440b98d80dc408eb292e1, dee122209eeb4c2281feaeb5eb5812bb; no voters:
I20260928 15:02:29.488677 10324 raft_consensus.cc:2781] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:29.488842 10324 raft_consensus.cc:482] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:29.488961 10324 raft_consensus.cc:3037] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:29.491134 10324 raft_consensus.cc:504] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.491770 10324 leader_election.cc:288] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [CANDIDATE]: Term 1 election: Requested vote from peers a5bf3506b59440b98d80dc408eb292e1 (127.9.12.195:46609), 5f623016996240e4b5f8031daa49cca8 (127.9.12.194:38223)
I20260928 15:02:29.492183 10120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65d4c5fa5c64532a2f8a1b8ad70cda8" candidate_uuid: "dee122209eeb4c2281feaeb5eb5812bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f623016996240e4b5f8031daa49cca8"
I20260928 15:02:29.492192 10253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65d4c5fa5c64532a2f8a1b8ad70cda8" candidate_uuid: "dee122209eeb4c2281feaeb5eb5812bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bf3506b59440b98d80dc408eb292e1"
I20260928 15:02:29.492419 10253 raft_consensus.cc:3037] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:29.492408 10120 raft_consensus.cc:3037] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:29.495268 10120 raft_consensus.cc:2454] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P 5f623016996240e4b5f8031daa49cca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee122209eeb4c2281feaeb5eb5812bb in term 1.
I20260928 15:02:29.495307 10253 raft_consensus.cc:2454] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P a5bf3506b59440b98d80dc408eb292e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee122209eeb4c2281feaeb5eb5812bb in term 1.
I20260928 15:02:29.495674 9920 leader_election.cc:302] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f623016996240e4b5f8031daa49cca8, dee122209eeb4c2281feaeb5eb5812bb; no voters:
I20260928 15:02:29.496001 10324 raft_consensus.cc:2781] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:29.496918 10324 raft_consensus.cc:686] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [term 1 LEADER]: Becoming Leader. State: Replica: dee122209eeb4c2281feaeb5eb5812bb, State: Running, Role: LEADER
I20260928 15:02:29.497448 10324 consensus_queue.cc:237] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } }
I20260928 15:02:29.502056 9841 catalog_manager.cc:5495] T a65d4c5fa5c64532a2f8a1b8ad70cda8 P dee122209eeb4c2281feaeb5eb5812bb reported cstate change: term changed from 0 to 1, leader changed from <none> to dee122209eeb4c2281feaeb5eb5812bb (127.9.12.193). New cstate: current_term: 1 leader_uuid: "dee122209eeb4c2281feaeb5eb5812bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bf3506b59440b98d80dc408eb292e1" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f623016996240e4b5f8031daa49cca8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee122209eeb4c2281feaeb5eb5812bb" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 42795 } health_report { overall_health: HEALTHY } } }
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:29.578544 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.563775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.9.12.193:42795)
0928 15:02:29.564503 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.564508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.564525 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:02:29.564590 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.564594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.564654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.564742 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.566035 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.566782 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.568069 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.568504 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.568605 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.572734 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.572749 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.572764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.572799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.575713 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.576444 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.576452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.576454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.576614 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.577010 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.577019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.577021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.577357 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.577669 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.577955 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.578176 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":462,"threads_started":1}
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:29.624692 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.610597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.9.12.193:42795)
0928 15:02:29.610914 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.610920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.611076 (+ 156us) server_negotiation.cc:407] Connection header received
0928 15:02:29.611230 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.611234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.611303 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.611391 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.612783 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.613492 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.614812 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.615220 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.615315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.619175 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.619180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.619183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.619217 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.622164 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.622740 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.622744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.622746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.622935 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.623360 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.623364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.623366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.623639 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.623831 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.624113 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.624376 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:29.671747 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.656382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.9.12.193:42795)
0928 15:02:29.656662 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.656670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.656916 (+ 246us) server_negotiation.cc:407] Connection header received
0928 15:02:29.657150 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.657154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.657259 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.657398 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.658977 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.659746 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.661158 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.661621 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.661756 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.665607 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.665611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.665615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.665648 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.668800 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.669591 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.669597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.669599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.669739 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.670161 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.670171 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.670174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.670454 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.670748 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.671087 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.671331 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 15:02:29.673607 9945 tablet_service.cc:2908] Rejecting scan request for tablet a65d4c5fa5c64532a2f8a1b8ad70cda8: Uninitialized: clean time has not yet been initialized
I20260928 15:02:29.676517 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 9902
I20260928 15:02:29.689483 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.677588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.9.12.193:42795)
0928 15:02:29.677836 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.677842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.678073 (+ 231us) server_negotiation.cc:407] Connection header received
0928 15:02:29.678198 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.678201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.678256 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.678333 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.679641 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.680441 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.681777 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.682187 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.682313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.684776 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.684780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.684783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.684811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.687146 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.687782 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.687786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.687788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.687894 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.688227 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.688230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.688232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.688522 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.688697 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.688998 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.689153 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:02:29.690135 10025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:29.782300 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 9902
I20260928 15:02:29.799489 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10036
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:29.813673 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.800314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.9.12.194:38223)
0928 15:02:29.800490 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.800494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.800512 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:02:29.800606 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.800610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.800673 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.800764 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.801962 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.802683 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.804035 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.804654 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.804755 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.808658 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.808670 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.808684 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.808715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.811271 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.811880 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.811887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.811889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.812008 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.812377 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.812384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.812390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.812765 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.812993 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.813245 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.813403 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:02:29.814333 10160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:29.901290 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10036
I20260928 15:02:29.917685 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10170
Sep 28 15:02:29 dist-test-slave-dmhs krb5kdc[9796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:29.933290 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.918462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.9.12.195:46609)
0928 15:02:29.918658 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.918663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.918682 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:02:29.918737 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.918741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.918805 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.918886 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.920044 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.920822 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.922043 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.922439 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.922525 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.927477 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.927488 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.927501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.927538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.930564 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.931234 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.931241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.931243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.931377 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.931764 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.931769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.931772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.932138 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.932504 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.932733 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.932941 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:02:29.933981 10293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:30.022274 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10170
I20260928 15:02:30.035728 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 9809
I20260928 15:02:30.036880 9871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:30.144368 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 9809
2026-09-28T15:02:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4472 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:30 dist-test-slave-dmhs krb5kdc[10350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:30 dist-test-slave-dmhs krb5kdc[10350](info): set up 2 sockets
Sep 28 15:02:30 dist-test-slave-dmhs krb5kdc[10350](info): commencing operation
krb5kdc: starting...
W20260928 15:02:31.232643 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:31 dist-test-slave-dmhs krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:31.415518 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45563
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:31.733139 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:31.733438 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:31.733533 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:31.740983 10363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:31.741094 10363 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:02:31.741145 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:31.741207 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:31.741258 10363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:31.741320 10363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:31.751329 10363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:45563
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:31.752624 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:31.753963 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:31.766227 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:31.766346 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:31.766314 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:31.767755 10363 server_base.cc:1034] running on GCE node
I20260928 15:02:31.768669 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:31.770219 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:31.771452 10363 hybrid_clock.cc:648] HybridClock initialized: now 1790607751771393 us; error 37 us; skew 500 ppm
I20260928 15:02:31.771773 10363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:31 dist-test-slave-dmhs krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:31.775111 10363 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:31.776757 10363 webserver.cc:460] Webserver started at http://127.9.12.254:39827/ using document root <none> and password file <none>
I20260928 15:02:31.777539 10363 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:31.777662 10363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:31.778026 10363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:31.781440 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "2c47ace6d2b447faa3b9502372c61b7f"
format_stamp: "Formatted at 2026-09-28 15:02:31 on dist-test-slave-dmhs"
server_key: "853ccd0184d0e88f366725e2a25f2a7b"
server_key_iv: "e49ee1f2456997d9f4fb4b4c61cc7a8b"
server_key_version: "encryptionkey@0"
I20260928 15:02:31.782298 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "2c47ace6d2b447faa3b9502372c61b7f"
format_stamp: "Formatted at 2026-09-28 15:02:31 on dist-test-slave-dmhs"
server_key: "853ccd0184d0e88f366725e2a25f2a7b"
server_key_iv: "e49ee1f2456997d9f4fb4b4c61cc7a8b"
server_key_version: "encryptionkey@0"
I20260928 15:02:31.787683 10363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:02:31.791224 10378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:31.792649 10363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:02:31.792829 10363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "2c47ace6d2b447faa3b9502372c61b7f"
format_stamp: "Formatted at 2026-09-28 15:02:31 on dist-test-slave-dmhs"
server_key: "853ccd0184d0e88f366725e2a25f2a7b"
server_key_iv: "e49ee1f2456997d9f4fb4b4c61cc7a8b"
server_key_version: "encryptionkey@0"
I20260928 15:02:31.793048 10363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:31.813297 10363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:31.817435 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:31.817723 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:31.839875 10363 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:45563
I20260928 15:02:31.839893 10439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:45563 every 8 connection(s)
I20260928 15:02:31.841414 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:31.841845 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10363
I20260928 15:02:31.842105 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:31.842953 9267 external_mini_cluster.cc:1428] Setting key af16e72baefac2a51c4d0fc888750051
I20260928 15:02:31.846590 10440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:02:31 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:31.859009 10440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: Bootstrap starting.
I20260928 15:02:31.863261 10440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:31.864543 10440 log.cc:824] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:31.867831 10440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: No bootstrap required, opened a new log
I20260928 15:02:31.868112 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:31.845300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.9.12.254:45563)
0928 15:02:31.846068 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:31.846089 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:31.846140 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:02:31.848501 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:31.848530 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:31.848997 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:31.849400 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:31.850821 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:31.852872 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:31.854065 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:31.855070 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:31.855243 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:31.859518 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:31.859530 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:31.859550 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:31.859593 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:31.863734 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:31.864440 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:31.864449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:31.864457 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:31.864676 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:31.865074 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:31.865082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:31.865087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:31.865672 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:31.865946 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:31.866379 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 15:02:31.866798 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260928 15:02:31.875864 10440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c47ace6d2b447faa3b9502372c61b7f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45563 } }
I20260928 15:02:31.876190 10440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:31.876307 10440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c47ace6d2b447faa3b9502372c61b7f, State: Initialized, Role: FOLLOWER
I20260928 15:02:31.876986 10440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c47ace6d2b447faa3b9502372c61b7f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45563 } }
I20260928 15:02:31.877276 10440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:31.877415 10440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:31.877614 10440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:31.879680 10440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c47ace6d2b447faa3b9502372c61b7f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45563 } }
I20260928 15:02:31.880194 10440 leader_election.cc:302] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c47ace6d2b447faa3b9502372c61b7f; no voters:
I20260928 15:02:31.880856 10440 leader_election.cc:288] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:31.881004 10445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:31.882045 10445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [term 1 LEADER]: Becoming Leader. State: Replica: 2c47ace6d2b447faa3b9502372c61b7f, State: Running, Role: LEADER
I20260928 15:02:31.882627 10445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c47ace6d2b447faa3b9502372c61b7f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45563 } }
I20260928 15:02:31.883090 10440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:31.885473 10446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c47ace6d2b447faa3b9502372c61b7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c47ace6d2b447faa3b9502372c61b7f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45563 } } }
I20260928 15:02:31.885555 10447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c47ace6d2b447faa3b9502372c61b7f. Latest consensus state: current_term: 1 leader_uuid: "2c47ace6d2b447faa3b9502372c61b7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c47ace6d2b447faa3b9502372c61b7f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45563 } } }
I20260928 15:02:31.885823 10446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:31.885897 10447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:31.888152 10451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:31.894590 10451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:31.904408 10451 catalog_manager.cc:1347] Generated new cluster ID: 1e9263e569244dd7be10681bb062779f
I20260928 15:02:31.904583 10451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:31.927613 10451 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:31.929440 10451 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:31.940400 10451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: Generated new TSK 0
I20260928 15:02:31.941295 10451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:31.985284 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45563
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:32.307876 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:32.308167 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:32.308358 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:32.315522 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:32.315622 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:32.315786 10466 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:32.315917 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:32.325721 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:32.327013 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:32.328397 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:32.341337 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.341413 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.341374 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:32.342631 10466 server_base.cc:1034] running on GCE node
I20260928 15:02:32.343223 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:32.344148 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:32.345369 10466 hybrid_clock.cc:648] HybridClock initialized: now 1790607752345308 us; error 45 us; skew 500 ppm
I20260928 15:02:32.345664 10466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:32 dist-test-slave-dmhs krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:32.348860 10466 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:32.350345 10466 webserver.cc:460] Webserver started at http://127.9.12.193:37551/ using document root <none> and password file <none>
I20260928 15:02:32.351023 10466 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:32.351140 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:32.351473 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:32.354393 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "76b87beed2c440dab5faad40170603b8"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-dmhs"
server_key: "c8a81f6e924dd1d87e103d04e38daec3"
server_key_iv: "0fa8537f7de0f09071fbf1b68025fc22"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.355218 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "76b87beed2c440dab5faad40170603b8"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-dmhs"
server_key: "c8a81f6e924dd1d87e103d04e38daec3"
server_key_iv: "0fa8537f7de0f09071fbf1b68025fc22"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.360447 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:02:32.363896 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.365228 10466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:32.365401 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "76b87beed2c440dab5faad40170603b8"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-dmhs"
server_key: "c8a81f6e924dd1d87e103d04e38daec3"
server_key_iv: "0fa8537f7de0f09071fbf1b68025fc22"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.365607 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:32.378345 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:32.382251 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:32.382508 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:32.385232 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:32.385334 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.385486 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:32.385623 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 15:02:32 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:32.410280 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.385791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:38723 (local address 127.9.12.254:45563)
0928 15:02:32.388486 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.388498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.388516 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:02:32.390898 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.390907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.391006 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.391134 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.393789 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.394769 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.397468 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.398183 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.398315 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.403468 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.403473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.403476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.403511 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.406141 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.407293 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.407297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.407299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.407418 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.407964 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.407967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.407969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.408222 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.408456 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.409545 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.409829 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2575,"thread_start_us":2490,"threads_started":1}
I20260928 15:02:32.412202 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.386528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.9.12.193:38723)
0928 15:02:32.387607 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:32.387665 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:32.390551 (+ 2886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:32.391589 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:32.391603 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:32.392638 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:32.393582 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.393613 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.394921 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.394929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.397320 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.397332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.398428 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.398445 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.398766 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.399645 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:32.399698 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:32.403228 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:32.406304 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.406318 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.406344 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.407161 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.407534 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.407541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.407548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.407847 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.408398 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:32.408411 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:32.409017 (+ 606us) client_negotiation.cc:769] Sending connection context
0928 15:02:32.409401 (+ 384us) client_negotiation.cc:240] Negotiation successful
0928 15:02:32.410117 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":514,"threads_started":1}
I20260928 15:02:32.432197 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.418185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:48033 (local address 127.9.12.254:45563)
0928 15:02:32.418390 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.418397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.419056 (+ 659us) server_negotiation.cc:407] Connection header received
0928 15:02:32.419191 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.419199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.419298 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.419425 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.420255 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.421745 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.423102 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.424012 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.424163 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.426317 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.426323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.426328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.426365 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.429650 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.430290 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.430295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.430299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.430449 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.430791 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.430796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.430800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.431069 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.431227 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.431616 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.431829 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:02:32.432281 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.418279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.9.12.193:48033)
0928 15:02:32.418962 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:32.418977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:32.419082 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:32.419439 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:32.419442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:32.419850 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:32.420125 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.420133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.421884 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.421888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.422973 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.422984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.424204 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.424208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.424668 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.425424 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:32.425443 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:32.426118 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:32.429782 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.429788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.429792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.430174 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.430553 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.430558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.430561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.430686 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.431183 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:32.431188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:32.431365 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 15:02:32.431511 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:02:32.431832 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260928 15:02:32.446655 10466 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:46753
I20260928 15:02:32.446697 10601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:46753 every 8 connection(s)
I20260928 15:02:32.447896 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:32.450455 10602 heartbeater.cc:347] Connected to a master server at 127.9.12.254:45563
I20260928 15:02:32.450891 10602 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:32.451788 10602 heartbeater.cc:511] Master 127.9.12.254:45563 requested a full tablet report, sending...
I20260928 15:02:32.454272 10395 ts_manager.cc:194] Registered new tserver with Master: 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753)
I20260928 15:02:32.455650 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10466
I20260928 15:02:32.455863 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:32.456487 10395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:38723
I20260928 15:02:32.456696 9267 external_mini_cluster.cc:1428] Setting key e2823544b867fbf2543a172ec9a784e9
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:32.495163 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45563
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:32.811127 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:32.811400 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:32.811497 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:32.818776 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:32.818889 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:32.818970 10606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:32.819048 10606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:32.829079 10606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:32.830312 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:32.831710 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:32.844955 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.844941 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.848601 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:32.850224 10606 server_base.cc:1034] running on GCE node
I20260928 15:02:32.850929 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:32.852173 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:32.853526 10606 hybrid_clock.cc:648] HybridClock initialized: now 1790607752853460 us; error 37 us; skew 500 ppm
I20260928 15:02:32.853865 10606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:32 dist-test-slave-dmhs krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:32.857650 10606 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:32.859133 10606 webserver.cc:460] Webserver started at http://127.9.12.194:37753/ using document root <none> and password file <none>
I20260928 15:02:32.859881 10606 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:32.860010 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:32.860388 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:32.863399 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "fe16e15071df4247bf87350cb4840f79"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-dmhs"
server_key: "82b69e284a8ecf54d76ec426fa424a0c"
server_key_iv: "0b7bfd96963d8a7bde9a0cad7ce59ca2"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.864226 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "fe16e15071df4247bf87350cb4840f79"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-dmhs"
server_key: "82b69e284a8ecf54d76ec426fa424a0c"
server_key_iv: "0b7bfd96963d8a7bde9a0cad7ce59ca2"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.869752 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:02:32.873416 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.874810 10606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:32.874975 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "fe16e15071df4247bf87350cb4840f79"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-dmhs"
server_key: "82b69e284a8ecf54d76ec426fa424a0c"
server_key_iv: "0b7bfd96963d8a7bde9a0cad7ce59ca2"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.875219 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:32.900818 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:32.905340 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:32.905625 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:32.908339 10606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:32.908454 10606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.908600 10606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:32.908669 10606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:32 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:32.934687 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.909158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:56231 (local address 127.9.12.254:45563)
0928 15:02:32.909360 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.909365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.910896 (+ 1531us) server_negotiation.cc:407] Connection header received
0928 15:02:32.913213 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.913219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.913344 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.913452 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.915956 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.917070 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.920460 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.920982 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.921146 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.926825 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.926836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.926841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.926878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.930571 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.931648 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.931656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.931662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.931831 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.932575 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.932583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.932587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.932995 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.933167 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.934085 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.934317 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:02:32.935781 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.909900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.9.12.194:56231)
0928 15:02:32.910731 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:32.910776 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:32.912870 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:32.913782 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:32.913799 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:32.914685 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:32.915744 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.915771 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.917263 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.917269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.920167 (+ 2898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.920186 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.921278 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.921289 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.921721 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.922679 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:32.922722 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:32.926585 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:32.930785 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.930798 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.930820 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.931515 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.931961 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.931967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.931973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.932229 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.933143 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:32.933151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:32.933631 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 15:02:32.933991 (+ 360us) client_negotiation.cc:240] Negotiation successful
0928 15:02:32.934597 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":360,"threads_started":1}
I20260928 15:02:32.956401 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.942159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.9.12.194:41897)
0928 15:02:32.942810 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:32.942833 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:32.943003 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:32.943423 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:32.943426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:32.943882 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:32.944459 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.944473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.945435 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.945442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.946416 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.946427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.947176 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.947182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.947343 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.948053 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:32.948077 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:32.949340 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:32.952785 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.952790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.952797 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.953402 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.953871 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.953877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.953881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.954054 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.954900 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:32.954908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:32.955091 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 15:02:32.955276 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 15:02:32.955643 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":378,"threads_started":1}
I20260928 15:02:32.956784 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.942151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:41897 (local address 127.9.12.254:45563)
0928 15:02:32.942394 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.942403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.942925 (+ 522us) server_negotiation.cc:407] Connection header received
0928 15:02:32.943163 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.943173 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.943277 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.943415 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.944616 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.945304 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.946552 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.946988 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.947122 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.949532 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.949537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.949540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.949619 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.952642 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.953593 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.953601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.953607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.953773 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.954229 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.954235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.954239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.954673 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.954833 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.956167 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.956435 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:02:32.962406 10606 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:43915
I20260928 15:02:32.962505 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:43915 every 8 connection(s)
I20260928 15:02:32.963634 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:32.965567 10741 heartbeater.cc:347] Connected to a master server at 127.9.12.254:45563
I20260928 15:02:32.965781 10741 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:32.966420 10741 heartbeater.cc:511] Master 127.9.12.254:45563 requested a full tablet report, sending...
I20260928 15:02:32.967671 10395 ts_manager.cc:194] Registered new tserver with Master: fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915)
I20260928 15:02:32.968562 10395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:56231
I20260928 15:02:32.973006 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10606
I20260928 15:02:32.973196 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:32.973603 9267 external_mini_cluster.cc:1428] Setting key a89cb40260a4e57efd44ee0cd0686026
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:33.012897 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45563
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:33.337718 10745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:33.338006 10745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:33.338114 10745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:33.345592 10745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:33.345709 10745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:33.345803 10745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:33.345883 10745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:33.355662 10745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:33.356925 10745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:33.358220 10745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:33.371614 10751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:33.371572 10750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:33.371656 10753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:33.372587 10745 server_base.cc:1034] running on GCE node
I20260928 15:02:33.373222 10745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:33.374076 10745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:33.375316 10745 hybrid_clock.cc:648] HybridClock initialized: now 1790607753375249 us; error 44 us; skew 500 ppm
I20260928 15:02:33.375624 10745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:33 dist-test-slave-dmhs krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:33.378841 10745 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:33.380353 10745 webserver.cc:460] Webserver started at http://127.9.12.195:43739/ using document root <none> and password file <none>
I20260928 15:02:33.381073 10745 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:33.381193 10745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:33.381511 10745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:33.384593 10745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "e7cd251126404906b2717d1063245884"
format_stamp: "Formatted at 2026-09-28 15:02:33 on dist-test-slave-dmhs"
server_key: "ebb481b68809241adfe99bcfe4f16326"
server_key_iv: "991e38a598a3227f7654fee60e935025"
server_key_version: "encryptionkey@0"
I20260928 15:02:33.385393 10745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "e7cd251126404906b2717d1063245884"
format_stamp: "Formatted at 2026-09-28 15:02:33 on dist-test-slave-dmhs"
server_key: "ebb481b68809241adfe99bcfe4f16326"
server_key_iv: "991e38a598a3227f7654fee60e935025"
server_key_version: "encryptionkey@0"
I20260928 15:02:33.390803 10745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:02:33.394243 10760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:33.395539 10745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:33.395720 10745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "e7cd251126404906b2717d1063245884"
format_stamp: "Formatted at 2026-09-28 15:02:33 on dist-test-slave-dmhs"
server_key: "ebb481b68809241adfe99bcfe4f16326"
server_key_iv: "991e38a598a3227f7654fee60e935025"
server_key_version: "encryptionkey@0"
I20260928 15:02:33.395941 10745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:33.412927 10745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:33.417313 10745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:33.417603 10745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:33.420207 10745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:33.420334 10745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 15:02:33.420502 10745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:33.420581 10745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:33 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:33.445832 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.420915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:59399 (local address 127.9.12.254:45563)
0928 15:02:33.421110 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.421119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.422582 (+ 1463us) server_negotiation.cc:407] Connection header received
0928 15:02:33.424804 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.424810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.424907 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.425022 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.427362 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.428490 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.431843 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.432421 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.432570 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.438432 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.438437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.438441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.438477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.441713 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.442836 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.442840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.442842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.443088 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.443722 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.443729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.443732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.444042 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.444188 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.445187 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.445453 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:02:33.447005 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.421633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.9.12.195:59399)
0928 15:02:33.422414 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.422455 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.424489 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.425268 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.425276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.426009 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:33.427172 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.427202 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.428675 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.428680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.431648 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.431667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.432706 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.432716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.433141 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.434059 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:33.434105 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:33.438178 (+ 4073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:33.441886 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:33.441900 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:33.441924 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:33.442717 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:33.443246 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:33.443251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:33.443255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:33.443540 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:33.444186 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:33.444194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:33.444717 (+ 523us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.445092 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.445681 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":342,"threads_started":1}
I20260928 15:02:33.459918 10602 heartbeater.cc:503] Master 127.9.12.254:45563 was elected leader, sending a full tablet report...
I20260928 15:02:33.482571 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.466074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:57691 (local address 127.9.12.254:45563)
0928 15:02:33.466233 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.466238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.467466 (+ 1228us) server_negotiation.cc:407] Connection header received
0928 15:02:33.467676 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.467681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.467757 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.467858 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.469536 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.470720 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.472878 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.473312 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.473450 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.475742 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.475748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.475753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.475793 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.479318 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.480308 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.480314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.480317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.480468 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.480833 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.480838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.480841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.481216 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.481371 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.481857 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.482139 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:02:33.482947 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.466187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.9.12.195:57691)
0928 15:02:33.467349 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.467366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.467541 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.467883 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.467888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.468503 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:33.469314 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.469333 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.471062 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.471069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.472720 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.472733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.473517 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.473525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.473687 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.474484 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:33.474508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:33.475560 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:33.479456 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:33.479462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:33.479466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:33.480137 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:33.480581 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:33.480587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:33.480591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:33.480720 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:33.481329 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:33.481338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:33.481607 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.481751 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.482174 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":492,"threads_started":1}
I20260928 15:02:33.492238 10745 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:39283
I20260928 15:02:33.492313 10880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:39283 every 8 connection(s)
I20260928 15:02:33.493582 10745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:33.495837 10881 heartbeater.cc:347] Connected to a master server at 127.9.12.254:45563
I20260928 15:02:33.496068 10881 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:33.496829 10881 heartbeater.cc:511] Master 127.9.12.254:45563 requested a full tablet report, sending...
I20260928 15:02:33.498194 10395 ts_manager.cc:194] Registered new tserver with Master: e7cd251126404906b2717d1063245884 (127.9.12.195:39283)
I20260928 15:02:33.498991 10395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:59399
I20260928 15:02:33.501753 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10745
I20260928 15:02:33.501919 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:33.502287 9267 external_mini_cluster.cc:1428] Setting key c19eab9ca2230e30f5c3b1e5cedb490c
I20260928 15:02:33.505113 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:33 dist-test-slave-dmhs krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:33 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:33.554123 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.538118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.9.12.254:45563)
0928 15:02:33.538351 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.538358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.538595 (+ 237us) server_negotiation.cc:407] Connection header received
0928 15:02:33.538755 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.538773 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.538848 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.538934 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.540475 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.541235 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.542539 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.542956 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.543047 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.547311 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.547316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.547319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.547356 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.551378 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.552256 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.552319 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.552325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.552481 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.552839 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.552842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.552844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.553141 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.553322 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.553575 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.553745 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:02:33.558373 10395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:33.565397 10395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:33.605229 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.577810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:46753 (local address 127.0.0.1:43440)
0928 15:02:33.592481 (+ 14671us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.592540 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.592856 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.596422 (+ 3566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.596429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.596462 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.597234 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.597250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.599670 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.599675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.602165 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.602181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.603982 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.603987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.604232 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.604359 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.604512 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.604715 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14229,"mutex_wait_us":4,"thread_start_us":15569,"threads_started":1}
I20260928 15:02:33.606304 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.578032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.9.12.193:46753)
0928 15:02:33.595744 (+ 17712us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.595752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.595777 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:33.595887 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.595897 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.596173 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.597460 (+ 1287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.597719 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.599482 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.602374 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.603762 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.605501 (+ 1739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.605711 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.605781 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.605987 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17555,"thread_start_us":17403,"threads_started":1}
I20260928 15:02:33.606945 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.578248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.9.12.194:43915)
0928 15:02:33.593898 (+ 15650us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.593912 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.593955 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:02:33.594090 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.594102 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.594488 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.594768 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.596627 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.598432 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.601864 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.603373 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.603488 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.603704 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.606440 (+ 2736us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.606625 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15395,"thread_start_us":15756,"threads_started":1}
I20260928 15:02:33.607147 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.578114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:43915 (local address 127.0.0.1:48446)
0928 15:02:33.593744 (+ 15630us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.593764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.593929 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.595767 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.595775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.595828 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.596391 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.596407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.598602 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.598607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.601632 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.601649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.603914 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.603921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.606119 (+ 2198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.606179 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.606331 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.606500 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15428,"mutex_wait_us":1882,"thread_start_us":11469,"threads_started":1}
I20260928 15:02:33.610503 10671 tablet_service.cc:1467] Processing CreateTablet for tablet f5334f68167b44ab88d2189093f2ff73 (DEFAULT_TABLE table=test-table [id=8f4cfcd9c4a84fa78d38b03561b4425f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.610543 10532 tablet_service.cc:1467] Processing CreateTablet for tablet f5334f68167b44ab88d2189093f2ff73 (DEFAULT_TABLE table=test-table [id=8f4cfcd9c4a84fa78d38b03561b4425f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.611847 10532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5334f68167b44ab88d2189093f2ff73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.611951 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5334f68167b44ab88d2189093f2ff73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.614168 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.594339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.9.12.195:39283)
0928 15:02:33.605324 (+ 10985us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.605332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.605370 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:02:33.605480 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.605489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.605757 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.606025 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.606903 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.609005 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.611316 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.612974 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.613119 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.613344 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.613642 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.613838 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10794,"thread_start_us":10541,"threads_started":1}
I20260928 15:02:33.614189 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.594492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:39283 (local address 127.0.0.1:41632)
0928 15:02:33.604581 (+ 10089us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.604599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.604707 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.605992 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.605998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.606042 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.606582 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.606595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.609190 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.609195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.611142 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.611161 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.613190 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.613196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.613373 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.613419 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.613588 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.613751 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9958,"mutex_wait_us":106,"thread_start_us":9815,"threads_started":1}
I20260928 15:02:33.617916 10811 tablet_service.cc:1467] Processing CreateTablet for tablet f5334f68167b44ab88d2189093f2ff73 (DEFAULT_TABLE table=test-table [id=8f4cfcd9c4a84fa78d38b03561b4425f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.619601 10811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5334f68167b44ab88d2189093f2ff73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.626438 10900 tablet_bootstrap.cc:492] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Bootstrap starting.
I20260928 15:02:33.629347 10900 tablet_bootstrap.cc:654] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.630553 10900 log.cc:824] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:33.631661 10901 tablet_bootstrap.cc:492] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Bootstrap starting.
I20260928 15:02:33.636384 10901 tablet_bootstrap.cc:654] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.637897 10901 log.cc:824] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:33.638960 10903 tablet_bootstrap.cc:492] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Bootstrap starting.
I20260928 15:02:33.639210 10900 tablet_bootstrap.cc:492] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: No bootstrap required, opened a new log
I20260928 15:02:33.639520 10900 ts_tablet_manager.cc:1397] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260928 15:02:33.641927 10901 tablet_bootstrap.cc:492] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: No bootstrap required, opened a new log
I20260928 15:02:33.642256 10903 tablet_bootstrap.cc:654] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.642289 10901 ts_tablet_manager.cc:1397] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 15:02:33.643137 10903 log.cc:824] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:33.645977 10903 tablet_bootstrap.cc:492] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: No bootstrap required, opened a new log
I20260928 15:02:33.646242 10903 ts_tablet_manager.cc:1397] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:02:33.650940 10900 raft_consensus.cc:348] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.651446 10900 raft_consensus.cc:374] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.651612 10900 raft_consensus.cc:729] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe16e15071df4247bf87350cb4840f79, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.652324 10900 consensus_queue.cc:260] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.653514 10741 heartbeater.cc:503] Master 127.9.12.254:45563 was elected leader, sending a full tablet report...
I20260928 15:02:33.653362 10901 raft_consensus.cc:348] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.653820 10901 raft_consensus.cc:374] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.653925 10901 raft_consensus.cc:729] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b87beed2c440dab5faad40170603b8, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.654580 10901 consensus_queue.cc:260] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.655000 10903 raft_consensus.cc:348] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.655432 10903 raft_consensus.cc:374] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.655557 10903 raft_consensus.cc:729] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cd251126404906b2717d1063245884, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.655880 10900 ts_tablet_manager.cc:1428] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260928 15:02:33.656219 10903 consensus_queue.cc:260] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.658152 10901 ts_tablet_manager.cc:1428] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260928 15:02:33.659551 10881 heartbeater.cc:503] Master 127.9.12.254:45563 was elected leader, sending a full tablet report...
I20260928 15:02:33.660220 10903 ts_tablet_manager.cc:1428] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260928 15:02:33.700551 10603 tablet.cc:2369] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:33.715552 10742 tablet.cc:2369] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:33.737761 10908 raft_consensus.cc:482] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:33.738157 10908 raft_consensus.cc:504] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.739864 10908 leader_election.cc:288] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753), fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915)
W20260928 15:02:33.745369 10882 tablet.cc:2369] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:33.746692 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.740060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:46753 (local address 127.9.12.195:36731)
0928 15:02:33.740388 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.740407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.740566 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.741073 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.741076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.741099 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.741281 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.741288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.742784 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.742789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.744495 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.744510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.745972 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.745978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.746177 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.746201 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.746298 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.746432 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 15:02:33.746686 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.740358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:36731 (local address 127.9.12.193:46753)
0928 15:02:33.740553 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.740558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.740579 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:02:33.740692 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.740697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.740806 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.740894 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.741431 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.742634 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.744646 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.745830 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.745985 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.746061 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.746333 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.746454 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:02:33.747608 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.740319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:43915 (local address 127.9.12.195:52199)
0928 15:02:33.741392 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.741412 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.741585 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.741915 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.741920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.741945 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.742289 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.742300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.743710 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.743714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.745534 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.745548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.746919 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.746925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.747095 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.747131 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.747227 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.747355 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":669,"threads_started":1}
I20260928 15:02:33.747625 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.740699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:52199 (local address 127.9.12.194:43915)
0928 15:02:33.740925 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.740931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.741498 (+ 567us) server_negotiation.cc:407] Connection header received
0928 15:02:33.741726 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.741729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.741798 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.741865 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.742442 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.743540 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.745696 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.746671 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.746853 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.746941 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.747265 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.747364 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:02:33.747992 10557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5334f68167b44ab88d2189093f2ff73" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b87beed2c440dab5faad40170603b8" is_pre_election: true
I20260928 15:02:33.748600 10557 raft_consensus.cc:2454] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 0.
I20260928 15:02:33.748809 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5334f68167b44ab88d2189093f2ff73" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe16e15071df4247bf87350cb4840f79" is_pre_election: true
I20260928 15:02:33.749367 10691 raft_consensus.cc:2454] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 0.
I20260928 15:02:33.749367 10761 leader_election.cc:302] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b87beed2c440dab5faad40170603b8, e7cd251126404906b2717d1063245884; no voters:
I20260928 15:02:33.749796 10908 raft_consensus.cc:2781] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:33.749990 10908 raft_consensus.cc:482] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:33.750131 10908 raft_consensus.cc:3037] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.752530 10908 raft_consensus.cc:504] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.753144 10908 leader_election.cc:288] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 election: Requested vote from peers 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753), fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915)
I20260928 15:02:33.753767 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5334f68167b44ab88d2189093f2ff73" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe16e15071df4247bf87350cb4840f79"
I20260928 15:02:33.753778 10557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5334f68167b44ab88d2189093f2ff73" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b87beed2c440dab5faad40170603b8"
I20260928 15:02:33.753996 10691 raft_consensus.cc:3037] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.754001 10557 raft_consensus.cc:3037] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.757633 10691 raft_consensus.cc:2454] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 1.
I20260928 15:02:33.757828 10557 raft_consensus.cc:2454] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 1.
I20260928 15:02:33.758216 10763 leader_election.cc:302] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7cd251126404906b2717d1063245884, fe16e15071df4247bf87350cb4840f79; no voters:
I20260928 15:02:33.758589 10908 raft_consensus.cc:2781] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:33.759054 10908 raft_consensus.cc:686] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 1 LEADER]: Becoming Leader. State: Replica: e7cd251126404906b2717d1063245884, State: Running, Role: LEADER
I20260928 15:02:33.759624 10908 consensus_queue.cc:237] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.764600 10394 catalog_manager.cc:5495] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7cd251126404906b2717d1063245884 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "e7cd251126404906b2717d1063245884" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:33.818974 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.813695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.9.12.195:39283)
0928 15:02:33.813873 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.813878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.813952 (+ 74us) server_negotiation.cc:407] Connection header received
0928 15:02:33.814240 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.814244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.814383 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.814537 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:33.814846 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.815648 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.816998 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.817447 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.817549 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.817969 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:33.818488 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:33.818595 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.818696 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.818761 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:02:33.827275 10691 raft_consensus.cc:1261] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Refusing update from remote peer e7cd251126404906b2717d1063245884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:33.827589 10557 raft_consensus.cc:1261] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Refusing update from remote peer e7cd251126404906b2717d1063245884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:33.828202 10910 consensus_queue.cc:1035] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:33.828640 10908 consensus_queue.cc:1035] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:33.840935 10915 mvcc.cc:205] Tried to move back new op lower bound from 7334329359667286016 to 7334329359403577344. Current Snapshot: MvccSnapshot[applied={T|T < 7334329359667286016}]
I20260928 15:02:33.843375 10914 mvcc.cc:205] Tried to move back new op lower bound from 7334329359667286016 to 7334329359403577344. Current Snapshot: MvccSnapshot[applied={T|T < 7334329359667286016}]
I20260928 15:02:33.844362 10913 mvcc.cc:205] Tried to move back new op lower bound from 7334329359667286016 to 7334329359403577344. Current Snapshot: MvccSnapshot[applied={T|T < 7334329359667286016}]
I20260928 15:02:33.867558 10395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:33.867830 10395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:33.871454 10395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: Sending DeleteTablet for 3 replicas of tablet f5334f68167b44ab88d2189093f2ff73
I20260928 15:02:33.872651 10811 tablet_service.cc:1514] Processing DeleteTablet for tablet f5334f68167b44ab88d2189093f2ff73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:33 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:41632
I20260928 15:02:33.872766 10671 tablet_service.cc:1514] Processing DeleteTablet for tablet f5334f68167b44ab88d2189093f2ff73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:33 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48446
I20260928 15:02:33.872895 10532 tablet_service.cc:1514] Processing DeleteTablet for tablet f5334f68167b44ab88d2189093f2ff73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:33 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:43440
I20260928 15:02:33.875049 10395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:33.876184 10395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:33.885162 10670 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6db673a6d94aaa841221fe70b63ea9 (DEFAULT_TABLE table=test-table [id=19e79e40738748dfa78875db483c5e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.885841 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6db673a6d94aaa841221fe70b63ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.885773 10810 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6db673a6d94aaa841221fe70b63ea9 (DEFAULT_TABLE table=test-table [id=19e79e40738748dfa78875db483c5e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.885934 10531 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6db673a6d94aaa841221fe70b63ea9 (DEFAULT_TABLE table=test-table [id=19e79e40738748dfa78875db483c5e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.886420 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6db673a6d94aaa841221fe70b63ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.886452 10531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6db673a6d94aaa841221fe70b63ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.893513 10903 tablet_bootstrap.cc:492] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Bootstrap starting.
I20260928 15:02:33.893713 10901 tablet_bootstrap.cc:492] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Bootstrap starting.
I20260928 15:02:33.894840 10900 tablet_bootstrap.cc:492] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Bootstrap starting.
I20260928 15:02:33.895303 10901 tablet_bootstrap.cc:654] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.895833 10903 tablet_bootstrap.cc:654] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.896919 10900 tablet_bootstrap.cc:654] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.897185 10924 tablet_replica.cc:330] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: stopping tablet replica
I20260928 15:02:33.897807 10924 raft_consensus.cc:2229] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:33.898427 10924 raft_consensus.cc:2258] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:33.901206 10924 ts_tablet_manager.cc:1905] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:33.908368 10923 tablet_replica.cc:330] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: stopping tablet replica
I20260928 15:02:33.908910 10923 raft_consensus.cc:2229] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:33.909588 10900 tablet_bootstrap.cc:492] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: No bootstrap required, opened a new log
I20260928 15:02:33.909703 10923 raft_consensus.cc:2258] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:33.909700 10903 tablet_bootstrap.cc:492] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: No bootstrap required, opened a new log
I20260928 15:02:33.909868 10900 ts_tablet_manager.cc:1397] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260928 15:02:33.910166 10903 ts_tablet_manager.cc:1397] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260928 15:02:33.911146 10903 raft_consensus.cc:348] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.911458 10901 tablet_bootstrap.cc:492] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: No bootstrap required, opened a new log
I20260928 15:02:33.911684 10901 ts_tablet_manager.cc:1397] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260928 15:02:33.911798 10925 tablet_replica.cc:330] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: stopping tablet replica
I20260928 15:02:33.912524 10925 raft_consensus.cc:2229] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:33.912653 10903 raft_consensus.cc:374] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.912801 10903 raft_consensus.cc:729] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cd251126404906b2717d1063245884, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.912699 10901 raft_consensus.cc:348] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.912955 10901 raft_consensus.cc:374] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.913086 10901 raft_consensus.cc:729] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b87beed2c440dab5faad40170603b8, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.913192 10903 consensus_queue.cc:260] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.913712 10925 raft_consensus.cc:2258] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:33.913655 10901 consensus_queue.cc:260] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.914108 10903 ts_tablet_manager.cc:1428] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 15:02:33.911684 10923 ts_tablet_manager.cc:1905] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:33.914291 10900 raft_consensus.cc:348] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.914624 10900 raft_consensus.cc:374] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.915271 10901 ts_tablet_manager.cc:1428] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:33.915661 10900 raft_consensus.cc:729] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe16e15071df4247bf87350cb4840f79, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.915978 10924 ts_tablet_manager.cc:1918] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:33.916134 10924 log.cc:1196] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/f5334f68167b44ab88d2189093f2ff73
I20260928 15:02:33.916745 10924 ts_tablet_manager.cc:1939] T f5334f68167b44ab88d2189093f2ff73 P fe16e15071df4247bf87350cb4840f79: Deleting consensus metadata
I20260928 15:02:33.916735 10900 consensus_queue.cc:260] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.917537 10900 ts_tablet_manager.cc:1428] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260928 15:02:33.918937 10381 catalog_manager.cc:4841] TS fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915): tablet f5334f68167b44ab88d2189093f2ff73 (table test-table [id=8f4cfcd9c4a84fa78d38b03561b4425f]) successfully deleted
I20260928 15:02:33.924005 10923 ts_tablet_manager.cc:1918] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:33.924484 10923 log.cc:1196] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/f5334f68167b44ab88d2189093f2ff73
I20260928 15:02:33.924692 10925 ts_tablet_manager.cc:1905] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:33.925256 10923 ts_tablet_manager.cc:1939] T f5334f68167b44ab88d2189093f2ff73 P e7cd251126404906b2717d1063245884: Deleting consensus metadata
I20260928 15:02:33.927552 10379 catalog_manager.cc:4841] TS e7cd251126404906b2717d1063245884 (127.9.12.195:39283): tablet f5334f68167b44ab88d2189093f2ff73 (table test-table [id=8f4cfcd9c4a84fa78d38b03561b4425f]) successfully deleted
I20260928 15:02:33.930816 10925 ts_tablet_manager.cc:1918] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:33.931001 10925 log.cc:1196] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/f5334f68167b44ab88d2189093f2ff73
I20260928 15:02:33.931442 10925 ts_tablet_manager.cc:1939] T f5334f68167b44ab88d2189093f2ff73 P 76b87beed2c440dab5faad40170603b8: Deleting consensus metadata
I20260928 15:02:33.932937 10379 catalog_manager.cc:4841] TS 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753): tablet f5334f68167b44ab88d2189093f2ff73 (table test-table [id=8f4cfcd9c4a84fa78d38b03561b4425f]) successfully deleted
I20260928 15:02:33.950441 10910 raft_consensus.cc:482] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:33.950766 10910 raft_consensus.cc:504] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.951591 10910 leader_election.cc:288] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753), fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915)
I20260928 15:02:33.952114 10557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6db673a6d94aaa841221fe70b63ea9" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b87beed2c440dab5faad40170603b8" is_pre_election: true
I20260928 15:02:33.952410 10557 raft_consensus.cc:2454] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 0.
I20260928 15:02:33.952479 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6db673a6d94aaa841221fe70b63ea9" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe16e15071df4247bf87350cb4840f79" is_pre_election: true
I20260928 15:02:33.952896 10682 raft_consensus.cc:2454] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 0.
I20260928 15:02:33.952849 10761 leader_election.cc:302] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b87beed2c440dab5faad40170603b8, e7cd251126404906b2717d1063245884; no voters:
I20260928 15:02:33.953338 10910 raft_consensus.cc:2781] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:33.953478 10910 raft_consensus.cc:482] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:33.953613 10910 raft_consensus.cc:3037] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.956975 10910 raft_consensus.cc:504] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.957666 10910 leader_election.cc:288] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 election: Requested vote from peers 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753), fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915)
I20260928 15:02:33.958153 10557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6db673a6d94aaa841221fe70b63ea9" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b87beed2c440dab5faad40170603b8"
I20260928 15:02:33.958372 10557 raft_consensus.cc:3037] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.960523 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6db673a6d94aaa841221fe70b63ea9" candidate_uuid: "e7cd251126404906b2717d1063245884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe16e15071df4247bf87350cb4840f79"
I20260928 15:02:33.960770 10682 raft_consensus.cc:3037] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.964187 10682 raft_consensus.cc:2454] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 1.
I20260928 15:02:33.965494 10557 raft_consensus.cc:2454] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7cd251126404906b2717d1063245884 in term 1.
I20260928 15:02:33.966112 10761 leader_election.cc:302] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b87beed2c440dab5faad40170603b8, e7cd251126404906b2717d1063245884; no voters:
I20260928 15:02:33.966482 10910 raft_consensus.cc:2781] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:33.966688 10910 raft_consensus.cc:686] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 1 LEADER]: Becoming Leader. State: Replica: e7cd251126404906b2717d1063245884, State: Running, Role: LEADER
I20260928 15:02:33.967010 10910 consensus_queue.cc:237] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:33.978371 10395 catalog_manager.cc:5495] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7cd251126404906b2717d1063245884 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "e7cd251126404906b2717d1063245884" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } health_report { overall_health: HEALTHY } } }
Sep 28 15:02:34 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:34.042519 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.023298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.9.12.254:45563)
0928 15:02:34.023495 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.023503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.024128 (+ 625us) server_negotiation.cc:407] Connection header received
0928 15:02:34.024310 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.024316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.024414 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.024546 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:34.025788 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.026921 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.028124 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.028878 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.029010 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.036569 (+ 7559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:34.036574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:34.036577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:34.036607 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:34.039435 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.040544 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.040547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.040549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.040675 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.041127 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.041130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.041132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.041399 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:34.041505 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.042075 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.042266 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":64}
I20260928 15:02:34.042997 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.023326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45563 (local address 127.0.0.1:36512)
0928 15:02:34.024014 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:34.024040 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:34.024163 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:34.024557 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:34.024560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:34.025350 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:34.025636 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.025645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.027045 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.027049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.027984 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.027995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.029047 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.029052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.029160 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.033218 (+ 4058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:34.033236 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:34.036363 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:34.039556 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:34.039571 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:34.039579 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:34.040419 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:34.040783 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:34.040788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:34.040791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:34.041020 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:34.041523 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:34.041532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:34.041747 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 15:02:34.042001 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 15:02:34.042442 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":118,"thread_start_us":393,"threads_started":1}
I20260928 15:02:34.048111 10395 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:36512:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:02:34.049259 10395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:34.056445 10532 tablet_service.cc:1467] Processing CreateTablet for tablet d958ad9c235344b68cf35bb893743a10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2830609c282f46e5a9c89846f2827902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:34.056607 10671 tablet_service.cc:1467] Processing CreateTablet for tablet d958ad9c235344b68cf35bb893743a10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2830609c282f46e5a9c89846f2827902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:34.056751 10811 tablet_service.cc:1467] Processing CreateTablet for tablet d958ad9c235344b68cf35bb893743a10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2830609c282f46e5a9c89846f2827902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:34.057057 10532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d958ad9c235344b68cf35bb893743a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:34.057173 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d958ad9c235344b68cf35bb893743a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:34.057261 10811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d958ad9c235344b68cf35bb893743a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:34.064634 10901 tablet_bootstrap.cc:492] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: Bootstrap starting.
I20260928 15:02:34.064901 10903 tablet_bootstrap.cc:492] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: Bootstrap starting.
I20260928 15:02:34.066856 10903 tablet_bootstrap.cc:654] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:34.066870 10901 tablet_bootstrap.cc:654] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:34.068048 10900 tablet_bootstrap.cc:492] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: Bootstrap starting.
I20260928 15:02:34.069584 10900 tablet_bootstrap.cc:654] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:34.072067 10903 tablet_bootstrap.cc:492] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: No bootstrap required, opened a new log
I20260928 15:02:34.072255 10903 ts_tablet_manager.cc:1397] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 15:02:34.072960 10903 raft_consensus.cc:348] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.073181 10903 raft_consensus.cc:374] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:34.073284 10903 raft_consensus.cc:729] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cd251126404906b2717d1063245884, State: Initialized, Role: FOLLOWER
I20260928 15:02:34.073621 10903 consensus_queue.cc:260] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.074309 10903 ts_tablet_manager.cc:1428] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:34.074316 10910 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.074625 10910 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.076543 10900 tablet_bootstrap.cc:492] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: No bootstrap required, opened a new log
I20260928 15:02:34.076706 10900 ts_tablet_manager.cc:1397] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 15:02:34.077085 10901 tablet_bootstrap.cc:492] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: No bootstrap required, opened a new log
I20260928 15:02:34.077242 10901 ts_tablet_manager.cc:1397] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260928 15:02:34.077595 10900 raft_consensus.cc:348] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.077840 10900 raft_consensus.cc:374] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:34.077941 10900 raft_consensus.cc:729] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe16e15071df4247bf87350cb4840f79, State: Initialized, Role: FOLLOWER
I20260928 15:02:34.078085 10901 raft_consensus.cc:348] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.078315 10901 raft_consensus.cc:374] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:34.078192 10900 consensus_queue.cc:260] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.078385 10901 raft_consensus.cc:729] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b87beed2c440dab5faad40170603b8, State: Initialized, Role: FOLLOWER
I20260928 15:02:34.078624 10901 consensus_queue.cc:260] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.078795 10900 ts_tablet_manager.cc:1428] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:34.079263 10901 ts_tablet_manager.cc:1428] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 15:02:34.079137 10907 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.079553 10907 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.079885 10906 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.080314 10906 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.361208 10906 raft_consensus.cc:482] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:34.361414 10906 raft_consensus.cc:504] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.362867 10906 leader_election.cc:288] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753), e7cd251126404906b2717d1063245884 (127.9.12.195:39283)
I20260928 15:02:34.370337 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.363347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:55689 (local address 127.9.12.193:46753)
0928 15:02:34.363822 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.363826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.364182 (+ 356us) server_negotiation.cc:407] Connection header received
0928 15:02:34.364381 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.364387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.364468 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.365131 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:34.365282 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.366207 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.368133 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.369486 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.369715 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.369809 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.370010 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.370120 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":253,"threads_started":1}
I20260928 15:02:34.370476 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.363131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:46753 (local address 127.9.12.194:55689)
0928 15:02:34.364010 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:34.364030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:34.364196 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:34.364702 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:34.364710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:34.364760 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:34.365014 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.365023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.366340 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.366346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.367951 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.367966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.369660 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.369665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.369845 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.369887 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:02:34.369990 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:02:34.370146 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":692,"threads_started":1}
I20260928 15:02:34.371274 10557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d958ad9c235344b68cf35bb893743a10" candidate_uuid: "fe16e15071df4247bf87350cb4840f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b87beed2c440dab5faad40170603b8" is_pre_election: true
I20260928 15:02:34.371572 10557 raft_consensus.cc:2454] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe16e15071df4247bf87350cb4840f79 in term 0.
I20260928 15:02:34.372095 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.364385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:39189 (local address 127.9.12.195:39283)
0928 15:02:34.364970 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.364976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.364997 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:02:34.365084 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.365091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.365173 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.365276 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:34.365807 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.366668 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.369934 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.371139 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.371301 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.371366 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.371629 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.371770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":346,"threads_started":1}
I20260928 15:02:34.372099 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.364231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:39283 (local address 127.9.12.194:39189)
0928 15:02:34.364728 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:34.364749 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:34.364861 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:34.365309 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:34.365316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:34.365346 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:34.365649 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.365659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.366867 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.366873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.369720 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.369734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.371309 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.371313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.371429 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.371480 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 15:02:34.371590 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:02:34.371721 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":215,"threads_started":1}
I20260928 15:02:34.372030 10622 leader_election.cc:302] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b87beed2c440dab5faad40170603b8, fe16e15071df4247bf87350cb4840f79; no voters:
I20260928 15:02:34.372587 10906 raft_consensus.cc:2781] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:34.372714 10906 raft_consensus.cc:482] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:34.372820 10906 raft_consensus.cc:3037] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:34.373188 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d958ad9c235344b68cf35bb893743a10" candidate_uuid: "fe16e15071df4247bf87350cb4840f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7cd251126404906b2717d1063245884" is_pre_election: true
I20260928 15:02:34.373524 10835 raft_consensus.cc:2454] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe16e15071df4247bf87350cb4840f79 in term 0.
I20260928 15:02:34.375077 10906 raft_consensus.cc:504] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.375628 10906 leader_election.cc:288] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [CANDIDATE]: Term 1 election: Requested vote from peers 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753), e7cd251126404906b2717d1063245884 (127.9.12.195:39283)
I20260928 15:02:34.376113 10557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d958ad9c235344b68cf35bb893743a10" candidate_uuid: "fe16e15071df4247bf87350cb4840f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b87beed2c440dab5faad40170603b8"
I20260928 15:02:34.376216 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d958ad9c235344b68cf35bb893743a10" candidate_uuid: "fe16e15071df4247bf87350cb4840f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7cd251126404906b2717d1063245884"
I20260928 15:02:34.376350 10557 raft_consensus.cc:3037] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:34.376475 10835 raft_consensus.cc:3037] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:34.378664 10835 raft_consensus.cc:2454] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe16e15071df4247bf87350cb4840f79 in term 1.
I20260928 15:02:34.378772 10910 consensus_queue.cc:1035] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:34.379191 10622 leader_election.cc:302] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7cd251126404906b2717d1063245884, fe16e15071df4247bf87350cb4840f79; no voters:
I20260928 15:02:34.379287 10557 raft_consensus.cc:2454] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe16e15071df4247bf87350cb4840f79 in term 1.
I20260928 15:02:34.379482 10906 raft_consensus.cc:2781] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:34.379948 10906 raft_consensus.cc:686] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [term 1 LEADER]: Becoming Leader. State: Replica: fe16e15071df4247bf87350cb4840f79, State: Running, Role: LEADER
I20260928 15:02:34.380429 10906 consensus_queue.cc:237] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } }
I20260928 15:02:34.382618 10943 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: TxnStatusTablet state changed. Reason: New leader fe16e15071df4247bf87350cb4840f79. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.383073 10943 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:34.385140 10946 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:02:34.384907 10393 catalog_manager.cc:5495] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe16e15071df4247bf87350cb4840f79 (127.9.12.194). New cstate: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:34.385429 10946 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:02:34.399475 10921 consensus_queue.cc:1035] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:34.466409 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.450770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:43915 (local address 127.0.0.1:32976)
0928 15:02:34.451150 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:34.451173 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:34.451381 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:34.458995 (+ 7614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:34.459002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:34.459019 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:34.459294 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.459309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.461186 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.461192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.462862 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.462880 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.463978 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.463985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.464183 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.464305 (+ 122us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:34.465743 (+ 1438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:34.465782 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:02:34.465867 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:02:34.466045 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 15:02:34.467620 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.450764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.9.12.194:43915)
0928 15:02:34.458440 (+ 7676us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.458447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.458475 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:02:34.458590 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.458596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.458721 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.459505 (+ 784us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:34.459825 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.461016 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.463074 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.463797 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.464520 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.464692 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:34.465571 (+ 879us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:34.466879 (+ 1308us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.466965 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.467053 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7500,"thread_start_us":7980,"threads_started":1}
I20260928 15:02:34.472681 10556 raft_consensus.cc:1261] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Refusing update from remote peer fe16e15071df4247bf87350cb4840f79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:34.472812 10835 raft_consensus.cc:1261] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Refusing update from remote peer fe16e15071df4247bf87350cb4840f79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:34.473620 10943 consensus_queue.cc:1035] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:34.473975 10906 consensus_queue.cc:1035] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:34.488298 10910 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: TxnStatusTablet state changed. Reason: New leader fe16e15071df4247bf87350cb4840f79. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.488619 10910 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.490717 10907 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: TxnStatusTablet state changed. Reason: New leader fe16e15071df4247bf87350cb4840f79. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.491047 10907 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.491091 10943 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.491374 10943 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:34.494454 10906 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.494900 10906 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P fe16e15071df4247bf87350cb4840f79: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:34.511672 10907 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.511945 10907 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P 76b87beed2c440dab5faad40170603b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.512804 10910 tablet_replica.cc:439] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe16e15071df4247bf87350cb4840f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b87beed2c440dab5faad40170603b8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46753 } } peers { permanent_uuid: "fe16e15071df4247bf87350cb4840f79" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 43915 } } peers { permanent_uuid: "e7cd251126404906b2717d1063245884" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 39283 } } }
I20260928 15:02:34.513087 10910 tablet_replica.cc:442] T d958ad9c235344b68cf35bb893743a10 P e7cd251126404906b2717d1063245884: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:34.536747 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.528144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:58025 (local address 127.9.12.194:43915)
0928 15:02:34.528371 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.528378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.529302 (+ 924us) server_negotiation.cc:407] Connection header received
0928 15:02:34.529485 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.529491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.529580 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.529683 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:34.530324 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.531504 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.534517 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.535238 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.535366 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.535968 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:34.536108 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:34.536200 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.536360 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.536445 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:02:34.536860 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.528118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:43915 (local address 127.9.12.195:58025)
0928 15:02:34.529187 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:34.529206 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:34.529351 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:34.529750 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:34.529765 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:34.529789 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:34.530155 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.530165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.531649 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.531654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.534379 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.534394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.535487 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.535501 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.535709 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.535779 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:34.536227 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:34.536238 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:02:34.536329 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:02:34.536473 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":757,"threads_started":1}
I20260928 15:02:34.582091 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.574191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:52739 (local address 127.9.12.195:39283)
0928 15:02:34.574362 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.574366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.575288 (+ 922us) server_negotiation.cc:407] Connection header received
0928 15:02:34.575505 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.575510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.575603 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.575692 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:34.576456 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.577711 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.579417 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.580544 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.580677 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.581212 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:34.581408 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:34.581494 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.581672 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.581750 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:02:34.582221 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.574038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:39283 (local address 127.9.12.194:52739)
0928 15:02:34.575191 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:34.575206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:34.575372 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:34.575712 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:34.575716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:34.575728 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:34.576253 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.576313 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.577832 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.577840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.579274 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:34.579288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.580687 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.580693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:34.580864 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.580967 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:34.581532 (+ 565us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:34.581545 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:02:34.581638 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:02:34.581775 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":720,"threads_started":1}
I20260928 15:02:34.618034 10393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:34.618337 10393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:34.621459 10393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c47ace6d2b447faa3b9502372c61b7f: Sending DeleteTablet for 3 replicas of tablet 8c6db673a6d94aaa841221fe70b63ea9
I20260928 15:02:34.622341 10532 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6db673a6d94aaa841221fe70b63ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:34 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:43440
I20260928 15:02:34.622427 10811 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6db673a6d94aaa841221fe70b63ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:34 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:41632
I20260928 15:02:34.626808 10671 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6db673a6d94aaa841221fe70b63ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:34 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48446
I20260928 15:02:34.638085 10968 tablet_replica.cc:330] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: stopping tablet replica
I20260928 15:02:34.638262 10970 tablet_replica.cc:330] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: stopping tablet replica
I20260928 15:02:34.638513 10968 raft_consensus.cc:2229] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:34.639032 10968 raft_consensus.cc:2258] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:34.639071 10970 raft_consensus.cc:2229] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:34.640015 10967 tablet_replica.cc:330] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: stopping tablet replica
I20260928 15:02:34.640302 10968 ts_tablet_manager.cc:1905] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:34.640779 10970 raft_consensus.cc:2258] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:34.641248 10967 raft_consensus.cc:2229] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:34.642951 10970 ts_tablet_manager.cc:1905] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:34.643352 10967 raft_consensus.cc:2258] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:34.644770 10967 ts_tablet_manager.cc:1905] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:34.648629 10968 ts_tablet_manager.cc:1918] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:34.648819 10968 log.cc:1196] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/8c6db673a6d94aaa841221fe70b63ea9
I20260928 15:02:34.649340 10968 ts_tablet_manager.cc:1939] T 8c6db673a6d94aaa841221fe70b63ea9 P e7cd251126404906b2717d1063245884: Deleting consensus metadata
I20260928 15:02:34.650606 10379 catalog_manager.cc:4841] TS e7cd251126404906b2717d1063245884 (127.9.12.195:39283): tablet 8c6db673a6d94aaa841221fe70b63ea9 (table test-table [id=19e79e40738748dfa78875db483c5e29]) successfully deleted
I20260928 15:02:34.658582 10970 ts_tablet_manager.cc:1918] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:34.658844 10970 log.cc:1196] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/8c6db673a6d94aaa841221fe70b63ea9
I20260928 15:02:34.658887 10967 ts_tablet_manager.cc:1918] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:34.659055 10967 log.cc:1196] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/8c6db673a6d94aaa841221fe70b63ea9
I20260928 15:02:34.659477 10970 ts_tablet_manager.cc:1939] T 8c6db673a6d94aaa841221fe70b63ea9 P fe16e15071df4247bf87350cb4840f79: Deleting consensus metadata
I20260928 15:02:34.659528 10967 ts_tablet_manager.cc:1939] T 8c6db673a6d94aaa841221fe70b63ea9 P 76b87beed2c440dab5faad40170603b8: Deleting consensus metadata
I20260928 15:02:34.660670 10379 catalog_manager.cc:4841] TS 76b87beed2c440dab5faad40170603b8 (127.9.12.193:46753): tablet 8c6db673a6d94aaa841221fe70b63ea9 (table test-table [id=19e79e40738748dfa78875db483c5e29]) successfully deleted
I20260928 15:02:34.660696 10381 catalog_manager.cc:4841] TS fe16e15071df4247bf87350cb4840f79 (127.9.12.194:43915): tablet 8c6db673a6d94aaa841221fe70b63ea9 (table test-table [id=19e79e40738748dfa78875db483c5e29]) successfully deleted
Sep 28 15:02:34 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:34.675348 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.658332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.9.12.193:46753)
0928 15:02:34.658590 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.658598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.661317 (+ 2719us) server_negotiation.cc:407] Connection header received
0928 15:02:34.661520 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.661525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.661604 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.661705 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:34.663108 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.663799 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.665085 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.665483 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.665606 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.669565 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:34.669578 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:34.669592 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:34.669631 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:34.672620 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.673290 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.673297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.673300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.673430 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.673812 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.673816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.673818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.674182 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:34.674499 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.674782 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.675022 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260928 15:02:34.676036 10597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35164
I20260928 15:02:34.695390 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.678938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.9.12.254:45563)
0928 15:02:34.679641 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.679645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.679660 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:34.679709 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.679712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.679860 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.679973 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:34.681975 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.683229 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.685071 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.685713 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.685811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.688616 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:34.688621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:34.688624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:34.688659 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:34.691716 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.692592 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.692596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.692634 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.692956 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.693470 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.693473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.693476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.694015 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:34.694273 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.694624 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.694873 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":399,"threads_started":1}
W20260928 15:02:34.696116 10393 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32980
I20260928 15:02:34.698740 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10466
I20260928 15:02:34.715040 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.699650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.9.12.193:46753)
0928 15:02:34.699866 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.699876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.700753 (+ 877us) server_negotiation.cc:407] Connection header received
0928 15:02:34.700945 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.700949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.701013 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.701098 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:34.702791 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.703504 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.705373 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.705766 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.705864 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.708976 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:34.708982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:34.708987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:34.709027 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:34.712153 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.712996 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.713000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.713002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.713143 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.713565 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.713568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.713570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.713847 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:34.714041 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.714376 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.714672 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:02:34.715806 10597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35172
W20260928 15:02:34.716481 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:34.716692 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10466
I20260928 15:02:34.734156 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10606
Sep 28 15:02:34 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:34.749691 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.734981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.9.12.194:43915)
0928 15:02:34.735159 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.735167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.735190 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:02:34.735404 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.735409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.735481 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.735603 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:34.736801 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.737556 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.738892 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.739470 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.739601 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.743519 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:34.743530 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:34.743542 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:34.743580 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:34.746666 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.747332 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.747342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.747350 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.747483 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.747863 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.747869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.747872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.748450 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:34.748790 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.749087 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.749342 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 15:02:34.750428 10736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32992
W20260928 15:02:34.751180 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:34.751365 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10606
I20260928 15:02:34.771426 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10745
Sep 28 15:02:34 dist-test-slave-dmhs krb5kdc[10350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:34.786952 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:34.772296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.9.12.195:39283)
0928 15:02:34.772480 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:34.772487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:34.772508 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:02:34.772692 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:34.772698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:34.772772 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:34.772884 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:34.774096 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.774899 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.776152 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:34.776687 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:34.776836 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:34.781050 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:34.781066 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:34.781081 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:34.781119 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:34.783924 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.784705 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.784712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.784714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.784847 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:34.785224 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:34.785228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:34.785229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:34.785594 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:34.785897 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:34.786333 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 15:02:34.786596 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 15:02:34.787700 10876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796
W20260928 15:02:34.788444 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:34.788592 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10745
I20260928 15:02:34.809712 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10363
I20260928 15:02:34.810870 10435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:34.927603 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10363
2026-09-28T15:02:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4782 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:35 dist-test-slave-dmhs krb5kdc[10979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:35 dist-test-slave-dmhs krb5kdc[10979](info): set up 2 sockets
Sep 28 15:02:35 dist-test-slave-dmhs krb5kdc[10979](info): commencing operation
krb5kdc: starting...
W20260928 15:02:36.016767 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:36 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:36.204092 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40221
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:36.526563 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:36.526898 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:36.527034 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:36.534353 10992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:36.534458 10992 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:02:36.534502 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:36.534585 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:36.534657 10992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:36.534754 10992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:36.544376 10992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:36.545653 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:36.547005 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:36.558972 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:36.559374 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:36.559451 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:36.560631 10992 server_base.cc:1034] running on GCE node
I20260928 15:02:36.561681 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:36.563654 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:36.564991 10992 hybrid_clock.cc:648] HybridClock initialized: now 1790607756564876 us; error 61 us; skew 500 ppm
I20260928 15:02:36.565446 10992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:36 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:36.569175 10992 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:36.570633 10992 webserver.cc:460] Webserver started at http://127.9.12.254:36445/ using document root <none> and password file <none>
I20260928 15:02:36.571291 10992 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:36.571378 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:36.571638 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:36.574592 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "4cf097a382d74937854d1e54914bd258"
format_stamp: "Formatted at 2026-09-28 15:02:36 on dist-test-slave-dmhs"
server_key: "8af6297a26fcd7ce3a0ccbd76f649476"
server_key_iv: "883ae446511610d9d54ab817ade4a638"
server_key_version: "encryptionkey@0"
I20260928 15:02:36.575338 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "4cf097a382d74937854d1e54914bd258"
format_stamp: "Formatted at 2026-09-28 15:02:36 on dist-test-slave-dmhs"
server_key: "8af6297a26fcd7ce3a0ccbd76f649476"
server_key_iv: "883ae446511610d9d54ab817ade4a638"
server_key_version: "encryptionkey@0"
I20260928 15:02:36.581096 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:02:36.584412 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:36.585690 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:36.585824 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "4cf097a382d74937854d1e54914bd258"
format_stamp: "Formatted at 2026-09-28 15:02:36 on dist-test-slave-dmhs"
server_key: "8af6297a26fcd7ce3a0ccbd76f649476"
server_key_iv: "883ae446511610d9d54ab817ade4a638"
server_key_version: "encryptionkey@0"
I20260928 15:02:36.585999 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:36.597160 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:36.600948 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:36.601155 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:36.622495 10992 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:40221
I20260928 15:02:36.622516 11068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:40221 every 8 connection(s)
I20260928 15:02:36.623862 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:36.627785 11069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.633344 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 10992
I20260928 15:02:36.633561 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:36.634197 9267 external_mini_cluster.cc:1428] Setting key a0dc03500cd6fde41026e1fd454ebe5c
I20260928 15:02:36.638684 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Bootstrap starting.
I20260928 15:02:36.642411 11069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.643603 11069 log.cc:824] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:36.646792 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: No bootstrap required, opened a new log
Sep 28 15:02:36 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:36.654937 11069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf097a382d74937854d1e54914bd258" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40221 } }
I20260928 15:02:36.655332 11069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.655450 11069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf097a382d74937854d1e54914bd258, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.656076 11069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf097a382d74937854d1e54914bd258" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40221 } }
I20260928 15:02:36.656349 11069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:36.656502 11069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:36.656697 11069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.657900 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.636155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.9.12.254:40221)
0928 15:02:36.637076 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.637089 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.637130 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:02:36.638748 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.638770 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.639192 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.639514 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:36.641463 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.642848 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.644643 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.645421 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.645550 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.650209 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:36.650220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:36.650234 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:36.650276 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:36.653678 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.654398 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.654405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.654411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.654582 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.654964 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.654967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.654969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.655618 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:36.655975 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.656375 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.656861 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":413,"threads_started":1}
I20260928 15:02:36.659147 11069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf097a382d74937854d1e54914bd258" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40221 } }
I20260928 15:02:36.659739 11069 leader_election.cc:302] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cf097a382d74937854d1e54914bd258; no voters:
I20260928 15:02:36.660432 11069 leader_election.cc:288] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:36.660545 11074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:36.661428 11074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [term 1 LEADER]: Becoming Leader. State: Replica: 4cf097a382d74937854d1e54914bd258, State: Running, Role: LEADER
I20260928 15:02:36.662001 11074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf097a382d74937854d1e54914bd258" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40221 } }
I20260928 15:02:36.662863 11069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:36.664933 11075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cf097a382d74937854d1e54914bd258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf097a382d74937854d1e54914bd258" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40221 } } }
I20260928 15:02:36.665033 11076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cf097a382d74937854d1e54914bd258. Latest consensus state: current_term: 1 leader_uuid: "4cf097a382d74937854d1e54914bd258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf097a382d74937854d1e54914bd258" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40221 } } }
I20260928 15:02:36.665879 11075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:36.666077 11076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:36.667657 11084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:36.673092 11084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:36.681988 11084 catalog_manager.cc:1347] Generated new cluster ID: 5079f96f464b4a39942b48ff6c2e6215
I20260928 15:02:36.682119 11084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:36.696846 11084 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:36.698047 11084 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:36.709438 11084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Generated new TSK 0
I20260928 15:02:36.710306 11084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:36.764384 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40221
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:37.081701 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:37.081985 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:37.082088 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:37.089216 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:37.089319 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:37.089412 11095 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:02:37.089464 11095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:37.089535 11095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:37.098929 11095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:37.100158 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:37.101490 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:37.114130 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:37.114102 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:37.114068 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:37.115281 11095 server_base.cc:1034] running on GCE node
I20260928 15:02:37.115871 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:37.116829 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:37.118046 11095 hybrid_clock.cc:648] HybridClock initialized: now 1790607757117997 us; error 44 us; skew 500 ppm
I20260928 15:02:37.118353 11095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:37 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:37.121373 11095 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:37.122749 11095 webserver.cc:460] Webserver started at http://127.9.12.193:36631/ using document root <none> and password file <none>
I20260928 15:02:37.123416 11095 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:37.123562 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:37.123862 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:37.126669 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "2cda83718c3d41efb8ac3335f015f864"
format_stamp: "Formatted at 2026-09-28 15:02:37 on dist-test-slave-dmhs"
server_key: "5ec10dd2578378e09719088c6699db3a"
server_key_iv: "3b78427a450321fc1013f15b8693e51c"
server_key_version: "encryptionkey@0"
I20260928 15:02:37.127385 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "2cda83718c3d41efb8ac3335f015f864"
format_stamp: "Formatted at 2026-09-28 15:02:37 on dist-test-slave-dmhs"
server_key: "5ec10dd2578378e09719088c6699db3a"
server_key_iv: "3b78427a450321fc1013f15b8693e51c"
server_key_version: "encryptionkey@0"
I20260928 15:02:37.132526 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:02:37.135701 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:37.136950 11095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:37.137118 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "2cda83718c3d41efb8ac3335f015f864"
format_stamp: "Formatted at 2026-09-28 15:02:37 on dist-test-slave-dmhs"
server_key: "5ec10dd2578378e09719088c6699db3a"
server_key_iv: "3b78427a450321fc1013f15b8693e51c"
server_key_version: "encryptionkey@0"
I20260928 15:02:37.137341 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:37.147152 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:37.150805 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:37.151087 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:37.153731 11095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:37.153875 11095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:37.154384 11095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:37.154508 11095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:37 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:37.181007 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.154114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:48451 (local address 127.9.12.254:40221)
0928 15:02:37.154326 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:37.154332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:37.155711 (+ 1379us) server_negotiation.cc:407] Connection header received
0928 15:02:37.158692 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:37.158696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:37.158766 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:37.158853 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:37.161509 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.162245 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.165504 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.166000 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.166142 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.170993 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:37.170999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:37.171002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:37.171036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:37.174235 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.175259 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.175265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.175269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.175456 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.177568 (+ 2112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.177576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.177581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.178019 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:37.178223 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:37.180455 (+ 2232us) server_negotiation.cc:299] Negotiation successful
0928 15:02:37.180695 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:02:37.181468 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.154574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.9.12.193:48451)
0928 15:02:37.155532 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:37.155581 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:37.158331 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:37.159158 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:37.159175 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:37.160139 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:37.161292 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.161323 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.162424 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.162433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.165339 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.165360 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.166265 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.166275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.166630 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.167495 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:37.167534 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:37.170736 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:37.174374 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.174383 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.174399 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.175116 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.175613 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.175622 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.175632 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.177373 (+ 1741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.178186 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:37.178200 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:37.178712 (+ 512us) client_negotiation.cc:769] Sending connection context
0928 15:02:37.179104 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 15:02:37.179759 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":460,"threads_started":1}
I20260928 15:02:37.204360 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.188777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.9.12.193:46325)
0928 15:02:37.190042 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:37.190060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:37.190214 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:37.190613 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:37.190616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:37.191032 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:37.191309 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.191318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.193009 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.193020 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.194621 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.194642 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.195857 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.195864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.196030 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.196691 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:37.196729 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:37.197878 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:37.201733 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.201738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.201742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.202165 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.202514 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.202520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.202529 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.202710 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.203456 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:37.203462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:37.203634 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 15:02:37.203745 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 15:02:37.203980 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":980,"threads_started":1}
I20260928 15:02:37.204680 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.188857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:46325 (local address 127.9.12.254:40221)
0928 15:02:37.189145 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:37.189152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:37.190131 (+ 979us) server_negotiation.cc:407] Connection header received
0928 15:02:37.190338 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:37.190346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:37.190468 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:37.190609 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:37.191442 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.192840 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.194855 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.195626 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.195782 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.198113 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:37.198122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:37.198130 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:37.198185 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:37.201587 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.202276 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.202284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.202288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.202406 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.202861 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.202867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.202870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.203304 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:37.203462 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:37.203912 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 15:02:37.204223 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 15:02:37.217265 11095 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:37635
I20260928 15:02:37.217303 11229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:37635 every 8 connection(s)
I20260928 15:02:37.218489 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:37.220520 11230 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40221
I20260928 15:02:37.220711 11230 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:37.221308 11230 heartbeater.cc:511] Master 127.9.12.254:40221 requested a full tablet report, sending...
I20260928 15:02:37.223155 11024 ts_manager.cc:194] Registered new tserver with Master: 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635)
I20260928 15:02:37.223241 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 11095
I20260928 15:02:37.223439 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:37.224049 9267 external_mini_cluster.cc:1428] Setting key 74eb27f87da952cabd3322a64cb3f110
I20260928 15:02:37.224759 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:48451
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:37.263360 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40221
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:37.579113 11234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:37.579432 11234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:37.579610 11234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:37.586788 11234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:37.586956 11234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:37.587103 11234 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:02:37.587186 11234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:37.587311 11234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:37.597146 11234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:37.598452 11234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:37.599799 11234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:37.612952 11239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:37.612967 11242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:37.613013 11240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:37.613603 11234 server_base.cc:1034] running on GCE node
I20260928 15:02:37.614303 11234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:37.615294 11234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:37.616513 11234 hybrid_clock.cc:648] HybridClock initialized: now 1790607757616453 us; error 80 us; skew 500 ppm
I20260928 15:02:37.616828 11234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:37 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:37.619961 11234 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:37.621407 11234 webserver.cc:460] Webserver started at http://127.9.12.194:34379/ using document root <none> and password file <none>
I20260928 15:02:37.622107 11234 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:37.622242 11234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:37.622562 11234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:37.625485 11234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "275431e5965d46d78c5ce963060b1c27"
format_stamp: "Formatted at 2026-09-28 15:02:37 on dist-test-slave-dmhs"
server_key: "327c3a55f2238b86b5f40d802c9fa1d2"
server_key_iv: "7c2f74ff21cb5c17603c36d4a6d7c77f"
server_key_version: "encryptionkey@0"
I20260928 15:02:37.626243 11234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "275431e5965d46d78c5ce963060b1c27"
format_stamp: "Formatted at 2026-09-28 15:02:37 on dist-test-slave-dmhs"
server_key: "327c3a55f2238b86b5f40d802c9fa1d2"
server_key_iv: "7c2f74ff21cb5c17603c36d4a6d7c77f"
server_key_version: "encryptionkey@0"
I20260928 15:02:37.631544 11234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:02:37.635134 11249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:37.636547 11234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:02:37.636720 11234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "275431e5965d46d78c5ce963060b1c27"
format_stamp: "Formatted at 2026-09-28 15:02:37 on dist-test-slave-dmhs"
server_key: "327c3a55f2238b86b5f40d802c9fa1d2"
server_key_iv: "7c2f74ff21cb5c17603c36d4a6d7c77f"
server_key_version: "encryptionkey@0"
I20260928 15:02:37.636931 11234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:37.655622 11234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:37.659579 11234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:37.659869 11234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:37.662489 11234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:37.662617 11234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:37.662742 11234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:37.662832 11234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:37 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:37.687136 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.663188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:48149 (local address 127.9.12.254:40221)
0928 15:02:37.663379 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:37.663385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:37.664886 (+ 1501us) server_negotiation.cc:407] Connection header received
0928 15:02:37.667131 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:37.667137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:37.667244 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:37.667391 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:37.669775 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.670800 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.673562 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.674494 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.674674 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.679109 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:37.679121 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:37.679128 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:37.679186 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:37.683375 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.684428 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.684432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.684434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.684593 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.685234 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.685238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.685240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.685519 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:37.685640 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:37.686699 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0928 15:02:37.686903 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:02:37.689474 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.663757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.9.12.194:48149)
0928 15:02:37.664696 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:37.664747 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:37.666779 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:37.667728 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:37.667739 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:37.668698 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:37.669589 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.669613 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.670975 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.670979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.673387 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.673407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.674731 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.674735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.674912 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.675660 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:37.675701 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:37.678775 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:37.683526 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.683532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.683546 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.684262 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.684751 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.684759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.684764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.685095 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.685661 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:37.685673 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:37.686253 (+ 580us) client_negotiation.cc:769] Sending connection context
0928 15:02:37.686650 (+ 397us) client_negotiation.cc:240] Negotiation successful
0928 15:02:37.687377 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":465,"threads_started":1}
I20260928 15:02:37.713198 11234 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:36233
I20260928 15:02:37.713199 11368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:36233 every 8 connection(s)
I20260928 15:02:37.714164 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.697250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:34741 (local address 127.9.12.254:40221)
0928 15:02:37.697449 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:37.697456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:37.698128 (+ 672us) server_negotiation.cc:407] Connection header received
0928 15:02:37.698326 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:37.698334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:37.698430 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:37.698533 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:37.700169 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.701282 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.703607 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.704178 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.704358 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.707187 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:37.707193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:37.707196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:37.707225 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:37.710809 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.711776 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.711781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.711783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.711898 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.712468 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.712477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.712480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.712948 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:37.713101 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:37.713742 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 15:02:37.713907 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:02:37.714375 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.697399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.9.12.194:34741)
0928 15:02:37.698025 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:37.698040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:37.698190 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:37.698601 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:37.698609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:37.699447 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:37.699988 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.700004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.701432 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.701441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.703465 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:37.703485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.704427 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.704437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:37.704700 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.705459 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:37.705491 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:37.707003 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:37.711064 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.711072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.711078 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.711662 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.712065 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:37.712073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:37.712079 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:37.712289 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:37.713223 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:37.713230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:37.713450 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 15:02:37.713624 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 15:02:37.713931 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":385,"threads_started":1}
I20260928 15:02:37.714875 11234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:37.717149 11369 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40221
I20260928 15:02:37.718247 11369 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:37.718886 11369 heartbeater.cc:511] Master 127.9.12.254:40221 requested a full tablet report, sending...
I20260928 15:02:37.720186 11023 ts_manager.cc:194] Registered new tserver with Master: 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:37.721077 11023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:48149
I20260928 15:02:37.721463 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 11234
I20260928 15:02:37.721647 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:37.722239 9267 external_mini_cluster.cc:1428] Setting key 1856107fd809a1ac9fde27aa06b58bf8
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:37.761349 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40221
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:38.079707 11373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:38.080008 11373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:38.080135 11373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:38.087607 11373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:38.087735 11373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:38.087883 11373 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:02:38.087963 11373 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:38.088091 11373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:38.097981 11373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:38.099396 11373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:38.101393 11373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:38.116380 11378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:38.116279 11379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:38.116601 11381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:38.117553 11373 server_base.cc:1034] running on GCE node
I20260928 15:02:38.118206 11373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:38.119168 11373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:38.120362 11373 hybrid_clock.cc:648] HybridClock initialized: now 1790607758120293 us; error 52 us; skew 500 ppm
I20260928 15:02:38.120746 11373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:38 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:38.124294 11373 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:38.126166 11373 webserver.cc:460] Webserver started at http://127.9.12.195:33427/ using document root <none> and password file <none>
I20260928 15:02:38.127094 11373 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:38.127239 11373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:38.127629 11373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:38.131906 11373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "3aa6beb4f44248c0af4effcfbf21499b"
format_stamp: "Formatted at 2026-09-28 15:02:38 on dist-test-slave-dmhs"
server_key: "6e4f60f82fe3a251dc08e5dfb304d863"
server_key_iv: "fb8bb58a143d22a0abf48c425cd24b7e"
server_key_version: "encryptionkey@0"
I20260928 15:02:38.132961 11373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "3aa6beb4f44248c0af4effcfbf21499b"
format_stamp: "Formatted at 2026-09-28 15:02:38 on dist-test-slave-dmhs"
server_key: "6e4f60f82fe3a251dc08e5dfb304d863"
server_key_iv: "fb8bb58a143d22a0abf48c425cd24b7e"
server_key_version: "encryptionkey@0"
I20260928 15:02:38.140311 11373 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260928 15:02:38.144167 11388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:38.145678 11373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:38.145864 11373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "3aa6beb4f44248c0af4effcfbf21499b"
format_stamp: "Formatted at 2026-09-28 15:02:38 on dist-test-slave-dmhs"
server_key: "6e4f60f82fe3a251dc08e5dfb304d863"
server_key_iv: "fb8bb58a143d22a0abf48c425cd24b7e"
server_key_version: "encryptionkey@0"
I20260928 15:02:38.146109 11373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:38.182111 11373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:38.186964 11373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:38.187273 11373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:38.190407 11373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:38.190608 11373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:02:38.190760 11373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:38.190874 11373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:38 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:38.217155 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.192689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:38571 (local address 127.9.12.254:40221)
0928 15:02:38.192900 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.192908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.193941 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 15:02:38.197571 (+ 3630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.197581 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.197699 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.197798 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:38.201104 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.202045 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.204718 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.205290 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.205418 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.209828 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:38.209833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:38.209836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:38.209871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:38.213243 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.214264 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.214273 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.214284 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.214476 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.215124 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.215134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.215136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.215406 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:38.215561 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.216557 (+ 996us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.216785 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:02:38.218703 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.191978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.9.12.195:38571)
0928 15:02:38.193753 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.193812 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.197152 (+ 3340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.198426 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.198441 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.199446 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:38.200858 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.200901 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.202233 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.202239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.204578 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.204590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.205559 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.205567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.205861 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.206673 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:38.206721 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:38.209575 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:38.213445 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:38.213458 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:38.213478 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:38.214103 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:38.214631 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:38.214640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:38.214645 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:38.214954 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:38.215557 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:38.215565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:38.216057 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.216458 (+ 401us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.217046 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"thread_start_us":1153,"threads_started":1}
I20260928 15:02:38.227545 11230 heartbeater.cc:503] Master 127.9.12.254:40221 was elected leader, sending a full tablet report...
I20260928 15:02:38.243772 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.227410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:52933 (local address 127.9.12.254:40221)
0928 15:02:38.227596 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.227602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.230182 (+ 2580us) server_negotiation.cc:407] Connection header received
0928 15:02:38.230378 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.230383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.230463 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.230559 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:38.231520 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.232245 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.233540 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.234150 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.234285 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.236691 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:38.236697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:38.236702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:38.236744 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:38.240513 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.241722 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.241732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.241735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.241881 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.242309 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.242312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.242314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.242571 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:38.242729 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.243162 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.243382 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:02:38.243865 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.227438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.9.12.195:52933)
0928 15:02:38.230074 (+ 2636us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.230090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.230246 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.230589 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.230593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.231076 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:38.231371 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.231380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.232422 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.232429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.233398 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.233410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.234391 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.234398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.234566 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.235210 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:38.235228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:38.236437 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:38.240705 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:38.240711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:38.240720 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:38.241569 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:38.242016 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:38.242021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:38.242025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:38.242197 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:38.242691 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:38.242696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:38.242892 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.243028 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.243377 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2511,"thread_start_us":1967,"threads_started":1}
I20260928 15:02:38.245255 11373 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:43405
I20260928 15:02:38.245289 11507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:43405 every 8 connection(s)
I20260928 15:02:38.246690 11373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:38.248808 11508 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40221
I20260928 15:02:38.249050 11508 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:38.249673 11508 heartbeater.cc:511] Master 127.9.12.254:40221 requested a full tablet report, sending...
I20260928 15:02:38.250945 11023 ts_manager.cc:194] Registered new tserver with Master: 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405)
I20260928 15:02:38.251760 11023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:38571
I20260928 15:02:38.254670 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 11373
I20260928 15:02:38.254805 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:38.255242 9267 external_mini_cluster.cc:1428] Setting key 44654ad205c9887bf622cff5992ef249
I20260928 15:02:38.257979 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:38 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:38 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:38.304440 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.291186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.9.12.254:40221)
0928 15:02:38.291360 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.291365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.291672 (+ 307us) server_negotiation.cc:407] Connection header received
0928 15:02:38.291918 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.291925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.292015 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.292119 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:38.293617 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.294348 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.295490 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.295905 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.295991 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.299693 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:38.299698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:38.299702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:38.299738 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:38.302300 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.302836 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.302840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.302843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.302962 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.303271 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.303274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.303276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.303531 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:38.303702 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.303953 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.304126 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:02:38.308485 11023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:38.315224 11023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:38.337103 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.326700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.9.12.195:43405)
0928 15:02:38.327482 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.327493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.327530 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:02:38.327650 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.327656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.327923 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.328118 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:38.329580 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.331463 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.334770 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.335861 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.335989 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.336239 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.336549 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.336743 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":412,"threads_started":1}
I20260928 15:02:38.337150 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.326397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43405 (local address 127.0.0.1:41560)
0928 15:02:38.327230 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.327274 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.327460 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.328703 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.328714 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.328774 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:38.329284 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.329301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.331658 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.331664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.334580 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.334594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.336055 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.336061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.336299 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.336355 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.336489 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.336693 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":64,"thread_start_us":392,"threads_started":1}
I20260928 15:02:38.338253 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.327646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37635 (local address 127.0.0.1:35108)
0928 15:02:38.328191 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.328209 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.328353 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.330303 (+ 1950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.330307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.330332 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:38.330835 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.330854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.332566 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.332571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.334927 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.334941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.337111 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.337119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.337638 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.337673 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.337757 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.337882 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":271,"threads_started":1}
I20260928 15:02:38.338730 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.327909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.9.12.193:37635)
0928 15:02:38.329866 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.329872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.329895 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:02:38.329968 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.329973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.330129 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.330314 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:38.331078 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.332438 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.335077 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.336625 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.336747 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.336955 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.338304 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.338496 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1823,"thread_start_us":1688,"threads_started":1}
I20260928 15:02:38.339056 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.328577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.9.12.194:36233)
0928 15:02:38.329085 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.329093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.331250 (+ 2157us) server_negotiation.cc:407] Connection header received
0928 15:02:38.331353 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.331360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.331599 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.331776 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:38.332486 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.333394 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.335917 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.337236 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.337373 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.337547 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.338590 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.338778 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":415,"threads_started":1}
I20260928 15:02:38.339200 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.328868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36233 (local address 127.0.0.1:57184)
0928 15:02:38.330994 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.331012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.331120 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.331780 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.331790 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.331818 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:38.332253 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.332287 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.333534 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.333542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.335746 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.335761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.337690 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.337698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.337885 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.337918 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.338012 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.338139 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1991,"thread_start_us":1830,"threads_started":1}
I20260928 15:02:38.341681 11438 tablet_service.cc:1467] Processing CreateTablet for tablet e3d9c745328e4df0ac88e9d3c692fc14 (DEFAULT_TABLE table=test-table [id=e8e68f31c5b0487080a894d5f7bfae14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:38.342361 11160 tablet_service.cc:1467] Processing CreateTablet for tablet e3d9c745328e4df0ac88e9d3c692fc14 (DEFAULT_TABLE table=test-table [id=e8e68f31c5b0487080a894d5f7bfae14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:38.342741 11299 tablet_service.cc:1467] Processing CreateTablet for tablet e3d9c745328e4df0ac88e9d3c692fc14 (DEFAULT_TABLE table=test-table [id=e8e68f31c5b0487080a894d5f7bfae14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:38.343263 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d9c745328e4df0ac88e9d3c692fc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.343721 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d9c745328e4df0ac88e9d3c692fc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.343920 11299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d9c745328e4df0ac88e9d3c692fc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.356081 11527 tablet_bootstrap.cc:492] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Bootstrap starting.
I20260928 15:02:38.359723 11527 tablet_bootstrap.cc:654] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.361004 11529 tablet_bootstrap.cc:492] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Bootstrap starting.
I20260928 15:02:38.361001 11527 log.cc:824] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:38.363934 11528 tablet_bootstrap.cc:492] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Bootstrap starting.
I20260928 15:02:38.365648 11529 tablet_bootstrap.cc:654] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.366971 11529 log.cc:824] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:38.367340 11528 tablet_bootstrap.cc:654] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.368280 11527 tablet_bootstrap.cc:492] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: No bootstrap required, opened a new log
I20260928 15:02:38.368396 11528 log.cc:824] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:38.368646 11527 ts_tablet_manager.cc:1397] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260928 15:02:38.371117 11529 tablet_bootstrap.cc:492] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: No bootstrap required, opened a new log
I20260928 15:02:38.371366 11529 ts_tablet_manager.cc:1397] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260928 15:02:38.373468 11528 tablet_bootstrap.cc:492] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: No bootstrap required, opened a new log
I20260928 15:02:38.373719 11528 ts_tablet_manager.cc:1397] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 15:02:38.379571 11527 raft_consensus.cc:348] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.380023 11527 raft_consensus.cc:374] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.380178 11527 raft_consensus.cc:729] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275431e5965d46d78c5ce963060b1c27, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.380450 11528 raft_consensus.cc:348] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.380847 11528 raft_consensus.cc:374] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.380946 11528 raft_consensus.cc:729] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.380867 11527 consensus_queue.cc:260] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.381616 11528 consensus_queue.cc:260] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.381979 11529 raft_consensus.cc:348] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.382457 11529 raft_consensus.cc:374] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.382597 11529 raft_consensus.cc:729] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cda83718c3d41efb8ac3335f015f864, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.382649 11527 ts_tablet_manager.cc:1428] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 15:02:38.383324 11529 consensus_queue.cc:260] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.383680 11369 heartbeater.cc:503] Master 127.9.12.254:40221 was elected leader, sending a full tablet report...
I20260928 15:02:38.384202 11508 heartbeater.cc:503] Master 127.9.12.254:40221 was elected leader, sending a full tablet report...
I20260928 15:02:38.385408 11529 ts_tablet_manager.cc:1428] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 15:02:38.386068 11528 ts_tablet_manager.cc:1428] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260928 15:02:38.468415 11370 tablet.cc:2369] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:38.470763 11231 tablet.cc:2369] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:38.498639 11509 tablet.cc:2369] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:38.541117 11534 raft_consensus.cc:482] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:38.541494 11534 raft_consensus.cc:504] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.543198 11534 leader_election.cc:288] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:38.549268 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.543549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37635 (local address 127.9.12.195:38439)
0928 15:02:38.543752 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.543769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.543873 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.544269 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.544300 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.544333 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:38.544610 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.544637 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.545842 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.545846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.547545 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.547561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.548751 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.548754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.548886 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.548910 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.548982 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.549073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 15:02:38.549396 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.543566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:38439 (local address 127.9.12.193:37635)
0928 15:02:38.543799 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.543804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.543829 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:38.544045 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.544051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.544133 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.544235 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:38.544778 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.545715 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.547706 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.548623 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.548779 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.548851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.549029 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.549155 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:02:38.550418 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.544000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36233 (local address 127.9.12.195:60807)
0928 15:02:38.544818 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.544830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.544922 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.545221 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.545224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.545242 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:38.545508 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.545514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.546568 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.546573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.548234 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.548245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.549820 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.549826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.549981 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.550003 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.550108 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.550201 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":487,"threads_started":1}
I20260928 15:02:38.550545 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.544116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:60807 (local address 127.9.12.194:36233)
0928 15:02:38.544319 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.544324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.544911 (+ 587us) server_negotiation.cc:407] Connection header received
0928 15:02:38.545017 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.545022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.545109 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.545203 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:38.545652 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.546428 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.548421 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.549610 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.549811 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.549877 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.550158 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.550292 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:02:38.550420 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9c745328e4df0ac88e9d3c692fc14" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864" is_pre_election: true
I20260928 15:02:38.550987 11185 raft_consensus.cc:2454] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:38.551664 11391 leader_election.cc:302] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cda83718c3d41efb8ac3335f015f864, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:38.551600 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9c745328e4df0ac88e9d3c692fc14" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27" is_pre_election: true
I20260928 15:02:38.552047 11534 raft_consensus.cc:2781] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:38.552114 11323 raft_consensus.cc:2454] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:38.552217 11534 raft_consensus.cc:482] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:38.552346 11534 raft_consensus.cc:3037] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.554554 11534 raft_consensus.cc:504] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.555050 11534 leader_election.cc:288] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Requested vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:38.555491 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9c745328e4df0ac88e9d3c692fc14" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864"
I20260928 15:02:38.555521 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9c745328e4df0ac88e9d3c692fc14" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27"
I20260928 15:02:38.555692 11185 raft_consensus.cc:3037] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.555703 11323 raft_consensus.cc:3037] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.557916 11185 raft_consensus.cc:2454] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:38.558069 11323 raft_consensus.cc:2454] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:38.558311 11391 leader_election.cc:302] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cda83718c3d41efb8ac3335f015f864, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:38.558588 11534 raft_consensus.cc:2781] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:38.559019 11534 raft_consensus.cc:686] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Becoming Leader. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Running, Role: LEADER
I20260928 15:02:38.559499 11534 consensus_queue.cc:237] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } }
I20260928 15:02:38.563956 11024 catalog_manager.cc:5495] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195). New cstate: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:02:38 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:38.611303 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.598595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40221 (local address 127.0.0.1:50188)
0928 15:02:38.599142 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.599156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.599281 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.599579 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.599582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.600221 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:38.600546 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.600556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.601544 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.601547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.602505 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.602516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.603171 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.603175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.603295 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.603943 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:38.603961 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:38.606143 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:38.609105 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:38.609116 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:38.609122 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:38.609547 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:38.609860 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:38.609863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:38.609865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:38.609991 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:38.610436 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:38.610442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:38.610579 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.610712 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.610912 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":285,"threads_started":1}
I20260928 15:02:38.611382 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.598530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.9.12.254:40221)
0928 15:02:38.598694 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.598698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.599244 (+ 546us) server_negotiation.cc:407] Connection header received
0928 15:02:38.599384 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.599388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.599448 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.599534 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:38.600680 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.601438 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.602618 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.603023 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.603111 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.606323 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:38.606328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:38.606331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:38.606359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:38.608975 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.609643 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.609648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.609650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.609772 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.610082 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.610085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.610087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.610327 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:38.610431 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.610797 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.611036 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":49}
I20260928 15:02:38.616194 11024 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:50188:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:02:38.617332 11024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:38.623972 11160 tablet_service.cc:1467] Processing CreateTablet for tablet a7c3052dff524eeeb76cf25e32244f47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f52641afe04412f98da5fb93fbd5ef5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:38.624917 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c3052dff524eeeb76cf25e32244f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.624801 11438 tablet_service.cc:1467] Processing CreateTablet for tablet a7c3052dff524eeeb76cf25e32244f47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f52641afe04412f98da5fb93fbd5ef5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:38.625716 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c3052dff524eeeb76cf25e32244f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.629814 11299 tablet_service.cc:1467] Processing CreateTablet for tablet a7c3052dff524eeeb76cf25e32244f47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f52641afe04412f98da5fb93fbd5ef5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:38.630738 11299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c3052dff524eeeb76cf25e32244f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.630780 11529 tablet_bootstrap.cc:492] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: Bootstrap starting.
I20260928 15:02:38.632455 11529 tablet_bootstrap.cc:654] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.633469 11528 tablet_bootstrap.cc:492] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: Bootstrap starting.
I20260928 15:02:38.635852 11528 tablet_bootstrap.cc:654] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.637434 11527 tablet_bootstrap.cc:492] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: Bootstrap starting.
I20260928 15:02:38.639452 11527 tablet_bootstrap.cc:654] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.640102 11529 tablet_bootstrap.cc:492] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: No bootstrap required, opened a new log
I20260928 15:02:38.640245 11529 ts_tablet_manager.cc:1397] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260928 15:02:38.641084 11528 tablet_bootstrap.cc:492] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: No bootstrap required, opened a new log
I20260928 15:02:38.641233 11528 ts_tablet_manager.cc:1397] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260928 15:02:38.641279 11529 raft_consensus.cc:348] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.641716 11529 raft_consensus.cc:374] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.641887 11529 raft_consensus.cc:729] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cda83718c3d41efb8ac3335f015f864, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.642299 11529 consensus_queue.cc:260] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.643072 11529 ts_tablet_manager.cc:1428] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:02:38.642931 11528 raft_consensus.cc:348] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.642910 11535 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.643316 11535 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.643159 11528 raft_consensus.cc:374] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.643436 11528 raft_consensus.cc:729] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.643663 11528 consensus_queue.cc:260] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.644205 11528 ts_tablet_manager.cc:1428] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260928 15:02:38.644054 11534 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.644366 11534 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.646529 11527 tablet_bootstrap.cc:492] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: No bootstrap required, opened a new log
I20260928 15:02:38.646679 11527 ts_tablet_manager.cc:1397] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260928 15:02:38.647403 11527 raft_consensus.cc:348] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.647641 11527 raft_consensus.cc:374] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.647725 11527 raft_consensus.cc:729] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275431e5965d46d78c5ce963060b1c27, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.647930 11527 consensus_queue.cc:260] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.648571 11527 ts_tablet_manager.cc:1428] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:02:38.648433 11533 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.648883 11533 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.729271 11534 raft_consensus.cc:482] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:38.729501 11534 raft_consensus.cc:504] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.730209 11534 leader_election.cc:288] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:38.730749 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c3052dff524eeeb76cf25e32244f47" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27" is_pre_election: true
I20260928 15:02:38.730719 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c3052dff524eeeb76cf25e32244f47" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864" is_pre_election: true
I20260928 15:02:38.730997 11323 raft_consensus.cc:2454] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:38.730997 11185 raft_consensus.cc:2454] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:38.731406 11391 leader_election.cc:302] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:38.731698 11534 raft_consensus.cc:2781] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:38.731806 11534 raft_consensus.cc:482] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:38.731930 11534 raft_consensus.cc:3037] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.733963 11534 raft_consensus.cc:504] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.734505 11534 leader_election.cc:288] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Requested vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:38.734959 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c3052dff524eeeb76cf25e32244f47" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864"
I20260928 15:02:38.735208 11185 raft_consensus.cc:3037] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.735159 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c3052dff524eeeb76cf25e32244f47" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27"
I20260928 15:02:38.735386 11323 raft_consensus.cc:3037] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.737785 11323 raft_consensus.cc:2454] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:38.738018 11185 raft_consensus.cc:2454] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:38.738204 11391 leader_election.cc:302] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:38.738459 11534 raft_consensus.cc:2781] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:38.738637 11534 raft_consensus.cc:686] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Becoming Leader. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Running, Role: LEADER
I20260928 15:02:38.738899 11534 consensus_queue.cc:237] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.740523 11537 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: TxnStatusTablet state changed. Reason: New leader 3aa6beb4f44248c0af4effcfbf21499b. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.740938 11537 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:38.741751 11549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:02:38.742048 11549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:02:38.742298 11024 catalog_manager.cc:5495] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195). New cstate: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:38.769341 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.762921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43405 (local address 127.0.0.1:41574)
0928 15:02:38.763146 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.763168 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.763298 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.763814 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.763817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.763827 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:38.764028 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.764035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.765430 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.765433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.766475 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.766486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.767540 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.767547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.767750 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.767809 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:38.768900 (+ 1091us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:38.768917 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.768974 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.769092 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:02:38.769397 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.762919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.9.12.195:43405)
0928 15:02:38.763104 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.763110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.763245 (+ 135us) server_negotiation.cc:407] Connection header received
0928 15:02:38.763499 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.763506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.763637 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.763804 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:38.764156 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.765311 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.766625 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.767370 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.767531 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.768012 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:38.768794 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:38.768957 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.769007 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.769104 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:02:38.777311 11185 raft_consensus.cc:1261] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:38.777308 11323 raft_consensus.cc:1261] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:38.778319 11534 consensus_queue.cc:1035] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:38.779028 11537 consensus_queue.cc:1035] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:38.789785 11535 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: TxnStatusTablet state changed. Reason: New leader 3aa6beb4f44248c0af4effcfbf21499b. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.790095 11535 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.790525 11533 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: TxnStatusTablet state changed. Reason: New leader 3aa6beb4f44248c0af4effcfbf21499b. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.790791 11533 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.794425 11534 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.794756 11534 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:38.795325 11534 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.795588 11534 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 3aa6beb4f44248c0af4effcfbf21499b: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:38.799484 11535 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.799764 11535 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 2cda83718c3d41efb8ac3335f015f864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.803305 11533 tablet_replica.cc:439] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } } }
I20260928 15:02:38.803570 11533 tablet_replica.cc:442] T a7c3052dff524eeeb76cf25e32244f47 P 275431e5965d46d78c5ce963060b1c27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:38.813169 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.807282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.9.12.195:43405)
0928 15:02:38.807465 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.807472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.807909 (+ 437us) server_negotiation.cc:407] Connection header received
0928 15:02:38.808062 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.808069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.808151 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.808249 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:38.808698 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.809887 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.811217 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.811898 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.812035 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.812433 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:38.812567 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:38.812650 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.812784 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.812860 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:02:38.830780 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.824402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:37793 (local address 127.9.12.195:43405)
0928 15:02:38.824549 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.824553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.825217 (+ 664us) server_negotiation.cc:407] Connection header received
0928 15:02:38.825379 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.825385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.825477 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.825571 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:38.826053 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.827210 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.828796 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.829479 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.829600 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.830008 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:38.830161 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:38.830263 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.830368 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.830452 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":46}
I20260928 15:02:38.830785 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.824480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43405 (local address 127.9.12.195:37793)
0928 15:02:38.825110 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:38.825125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:38.825253 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:38.825580 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:38.825583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:38.825591 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:38.825907 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.825914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.827367 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.827373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.828667 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:38.828678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.829644 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.829648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:38.829769 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.829817 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:38.830259 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:38.830266 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:02:38.830327 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 15:02:38.830419 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1,"thread_start_us":357,"threads_started":1}
I20260928 15:02:38.840530 11323 raft_consensus.cc:1261] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:38.840643 11185 raft_consensus.cc:1261] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:38.841144 11537 consensus_queue.cc:1035] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:38.841408 11534 consensus_queue.cc:1035] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:38.846787 11552 mvcc.cc:205] Tried to move back new op lower bound from 7334329380204453888 to 7334329379063693312. Current Snapshot: MvccSnapshot[applied={T|T < 7334329380204453888}]
I20260928 15:02:38.901294 11024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:38.901600 11024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:38.905165 11024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Sending DeleteTablet for 3 replicas of tablet e3d9c745328e4df0ac88e9d3c692fc14
I20260928 15:02:38.906298 11437 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d9c745328e4df0ac88e9d3c692fc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:38 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:41560
I20260928 15:02:38.906402 11299 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d9c745328e4df0ac88e9d3c692fc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:38 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:57184
I20260928 15:02:38.906416 11160 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d9c745328e4df0ac88e9d3c692fc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:38 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:35108
I20260928 15:02:38.909799 11024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:38.910455 11024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:38.917088 11571 tablet_replica.cc:330] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: stopping tablet replica
I20260928 15:02:38.917431 11569 tablet_replica.cc:330] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: stopping tablet replica
I20260928 15:02:38.917578 11571 raft_consensus.cc:2229] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:38.917960 11571 raft_consensus.cc:2258] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:38.917955 11569 raft_consensus.cc:2229] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:38.918781 11569 raft_consensus.cc:2258] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:38.921175 11569 ts_tablet_manager.cc:1905] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:38.921373 11298 tablet_service.cc:1467] Processing CreateTablet for tablet 492971010cd64eceb65c93864fb9a546 (DEFAULT_TABLE table=test-table [id=7a62c393253f44ecb7c9ca7e081f6151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:38.921948 11298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492971010cd64eceb65c93864fb9a546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.923206 11570 tablet_replica.cc:330] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: stopping tablet replica
I20260928 15:02:38.923744 11570 raft_consensus.cc:2229] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:38.924149 11570 raft_consensus.cc:2258] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:38.924844 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 492971010cd64eceb65c93864fb9a546 (DEFAULT_TABLE table=test-table [id=7a62c393253f44ecb7c9ca7e081f6151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:38.924901 11438 tablet_service.cc:1467] Processing CreateTablet for tablet 492971010cd64eceb65c93864fb9a546 (DEFAULT_TABLE table=test-table [id=7a62c393253f44ecb7c9ca7e081f6151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:38.925304 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492971010cd64eceb65c93864fb9a546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.925312 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492971010cd64eceb65c93864fb9a546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.928100 11570 ts_tablet_manager.cc:1905] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:38.931788 11529 tablet_bootstrap.cc:492] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Bootstrap starting.
I20260928 15:02:38.933645 11529 tablet_bootstrap.cc:654] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.933750 11571 ts_tablet_manager.cc:1905] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:38.935808 11569 ts_tablet_manager.cc:1918] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:38.939435 11527 tablet_bootstrap.cc:492] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Bootstrap starting.
I20260928 15:02:38.939656 11569 log.cc:1196] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/e3d9c745328e4df0ac88e9d3c692fc14
I20260928 15:02:38.940310 11569 ts_tablet_manager.cc:1939] T e3d9c745328e4df0ac88e9d3c692fc14 P 3aa6beb4f44248c0af4effcfbf21499b: Deleting consensus metadata
I20260928 15:02:38.941452 11527 tablet_bootstrap.cc:654] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.941788 11571 ts_tablet_manager.cc:1918] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:38.941978 11571 log.cc:1196] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/e3d9c745328e4df0ac88e9d3c692fc14
I20260928 15:02:38.942528 11571 ts_tablet_manager.cc:1939] T e3d9c745328e4df0ac88e9d3c692fc14 P 2cda83718c3d41efb8ac3335f015f864: Deleting consensus metadata
I20260928 15:02:38.942456 11010 catalog_manager.cc:4841] TS 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405): tablet e3d9c745328e4df0ac88e9d3c692fc14 (table test-table [id=e8e68f31c5b0487080a894d5f7bfae14]) successfully deleted
I20260928 15:02:38.943050 11570 ts_tablet_manager.cc:1918] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:38.943280 11570 log.cc:1196] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/e3d9c745328e4df0ac88e9d3c692fc14
I20260928 15:02:38.943874 11570 ts_tablet_manager.cc:1939] T e3d9c745328e4df0ac88e9d3c692fc14 P 275431e5965d46d78c5ce963060b1c27: Deleting consensus metadata
I20260928 15:02:38.944235 11010 catalog_manager.cc:4841] TS 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635): tablet e3d9c745328e4df0ac88e9d3c692fc14 (table test-table [id=e8e68f31c5b0487080a894d5f7bfae14]) successfully deleted
I20260928 15:02:38.945880 11010 catalog_manager.cc:4841] TS 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233): tablet e3d9c745328e4df0ac88e9d3c692fc14 (table test-table [id=e8e68f31c5b0487080a894d5f7bfae14]) successfully deleted
I20260928 15:02:38.946786 11528 tablet_bootstrap.cc:492] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Bootstrap starting.
I20260928 15:02:38.948625 11528 tablet_bootstrap.cc:654] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.948891 11527 tablet_bootstrap.cc:492] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: No bootstrap required, opened a new log
I20260928 15:02:38.949174 11529 tablet_bootstrap.cc:492] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: No bootstrap required, opened a new log
I20260928 15:02:38.949079 11527 ts_tablet_manager.cc:1397] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 15:02:38.949378 11529 ts_tablet_manager.cc:1397] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260928 15:02:38.950145 11529 raft_consensus.cc:348] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.950120 11527 raft_consensus.cc:348] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.950420 11527 raft_consensus.cc:374] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.950508 11527 raft_consensus.cc:729] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275431e5965d46d78c5ce963060b1c27, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.950531 11529 raft_consensus.cc:374] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.950736 11529 raft_consensus.cc:729] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cda83718c3d41efb8ac3335f015f864, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.950819 11527 consensus_queue.cc:260] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.951152 11529 consensus_queue.cc:260] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.951799 11529 ts_tablet_manager.cc:1428] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 15:02:38.952395 11527 ts_tablet_manager.cc:1428] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:38.955418 11528 tablet_bootstrap.cc:492] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: No bootstrap required, opened a new log
I20260928 15:02:38.955662 11528 ts_tablet_manager.cc:1397] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 15:02:38.956465 11528 raft_consensus.cc:348] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.956790 11528 raft_consensus.cc:374] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.956938 11528 raft_consensus.cc:729] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.957226 11528 consensus_queue.cc:260] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:38.957803 11528 ts_tablet_manager.cc:1428] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:39.051652 11537 raft_consensus.cc:482] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:39.051873 11537 raft_consensus.cc:504] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:39.052645 11537 leader_election.cc:288] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:39.053285 11322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492971010cd64eceb65c93864fb9a546" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27" is_pre_election: true
I20260928 15:02:39.053362 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492971010cd64eceb65c93864fb9a546" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864" is_pre_election: true
I20260928 15:02:39.053565 11322 raft_consensus.cc:2454] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:39.053650 11184 raft_consensus.cc:2454] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:39.054071 11391 leader_election.cc:302] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:39.054401 11537 raft_consensus.cc:2781] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:39.054538 11537 raft_consensus.cc:482] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:39.054661 11537 raft_consensus.cc:3037] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:39.056981 11537 raft_consensus.cc:504] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:39.057598 11537 leader_election.cc:288] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Requested vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:39.057993 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492971010cd64eceb65c93864fb9a546" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864"
I20260928 15:02:39.058100 11322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492971010cd64eceb65c93864fb9a546" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27"
I20260928 15:02:39.058199 11184 raft_consensus.cc:3037] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:39.058332 11322 raft_consensus.cc:3037] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:39.060406 11322 raft_consensus.cc:2454] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:39.060794 11391 leader_election.cc:302] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:39.061070 11537 raft_consensus.cc:2781] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:39.061228 11184 raft_consensus.cc:2454] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:39.061249 11537 raft_consensus.cc:686] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Becoming Leader. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Running, Role: LEADER
I20260928 15:02:39.061833 11537 consensus_queue.cc:237] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:39.064764 11024 catalog_manager.cc:5495] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195). New cstate: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:39.099498 11322 raft_consensus.cc:1261] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:39.099988 11184 raft_consensus.cc:1261] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:39.100090 11537 consensus_queue.cc:1035] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:39.100674 11534 consensus_queue.cc:1035] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:39.114418 11554 mvcc.cc:205] Tried to move back new op lower bound from 7334329381266100224 to 7334329381118791680. Current Snapshot: MvccSnapshot[applied={T|T < 7334329381266100224}]
W20260928 15:02:39.126591 11552 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:02:40.703936 11024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:40.704157 11024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:40.707875 11024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Sending DeleteTablet for 3 replicas of tablet 492971010cd64eceb65c93864fb9a546
I20260928 15:02:40.708613 11160 tablet_service.cc:1514] Processing DeleteTablet for tablet 492971010cd64eceb65c93864fb9a546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:40 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:35108
I20260928 15:02:40.708776 11298 tablet_service.cc:1514] Processing DeleteTablet for tablet 492971010cd64eceb65c93864fb9a546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:40 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:57184
I20260928 15:02:40.708811 11438 tablet_service.cc:1514] Processing DeleteTablet for tablet 492971010cd64eceb65c93864fb9a546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:40 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:41560
I20260928 15:02:40.720468 11598 tablet_replica.cc:330] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: stopping tablet replica
I20260928 15:02:40.720723 11598 raft_consensus.cc:2229] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:40.720978 11598 raft_consensus.cc:2258] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:40.722868 11598 ts_tablet_manager.cc:1905] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:40.727677 11598 ts_tablet_manager.cc:1918] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:40.727829 11598 log.cc:1196] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/492971010cd64eceb65c93864fb9a546
I20260928 15:02:40.728412 11598 ts_tablet_manager.cc:1939] T 492971010cd64eceb65c93864fb9a546 P 2cda83718c3d41efb8ac3335f015f864: Deleting consensus metadata
I20260928 15:02:40.728679 11600 tablet_replica.cc:330] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: stopping tablet replica
I20260928 15:02:40.728255 11599 tablet_replica.cc:330] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: stopping tablet replica
I20260928 15:02:40.729027 11600 raft_consensus.cc:2229] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:40.729315 11599 raft_consensus.cc:2229] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:40.729724 11010 catalog_manager.cc:4841] TS 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635): tablet 492971010cd64eceb65c93864fb9a546 (table test-table [id=7a62c393253f44ecb7c9ca7e081f6151]) successfully deleted
I20260928 15:02:40.730115 11599 raft_consensus.cc:2258] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:40.729796 11600 raft_consensus.cc:2258] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:40.731326 11600 ts_tablet_manager.cc:1905] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:40.731361 11599 ts_tablet_manager.cc:1905] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:40.738293 11600 ts_tablet_manager.cc:1918] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:40.738301 11599 ts_tablet_manager.cc:1918] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:40.738445 11599 log.cc:1196] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/492971010cd64eceb65c93864fb9a546
I20260928 15:02:40.738520 11600 log.cc:1196] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/492971010cd64eceb65c93864fb9a546
I20260928 15:02:40.739099 11599 ts_tablet_manager.cc:1939] T 492971010cd64eceb65c93864fb9a546 P 275431e5965d46d78c5ce963060b1c27: Deleting consensus metadata
I20260928 15:02:40.739133 11600 ts_tablet_manager.cc:1939] T 492971010cd64eceb65c93864fb9a546 P 3aa6beb4f44248c0af4effcfbf21499b: Deleting consensus metadata
I20260928 15:02:40.740338 11010 catalog_manager.cc:4841] TS 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233): tablet 492971010cd64eceb65c93864fb9a546 (table test-table [id=7a62c393253f44ecb7c9ca7e081f6151]) successfully deleted
I20260928 15:02:40.740669 11010 catalog_manager.cc:4841] TS 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405): tablet 492971010cd64eceb65c93864fb9a546 (table test-table [id=7a62c393253f44ecb7c9ca7e081f6151]) successfully deleted
Sep 28 15:02:40 dist-test-slave-dmhs krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:40 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:40.770766 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.756633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.9.12.254:40221)
0928 15:02:40.757250 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.757254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.757279 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:40.757450 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.757454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.757557 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.757661 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:40.758954 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.759742 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.761255 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.761662 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.761770 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.765878 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:40.765884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:40.765887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:40.765923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:40.768604 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:40.769194 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:40.769198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:40.769201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:40.769314 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:40.769613 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:40.769616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:40.769618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:40.769891 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:40.770114 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.770322 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.770490 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":342,"threads_started":1}
I20260928 15:02:40.774150 11024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:40.774791 11024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:40.781033 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 1e76c7f6ac6a477ba2af13b95c12858f (DEFAULT_TABLE table=test-table [id=2863872f3e004b9e94d579c3f45da060]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:40.781071 11298 tablet_service.cc:1467] Processing CreateTablet for tablet 1e76c7f6ac6a477ba2af13b95c12858f (DEFAULT_TABLE table=test-table [id=2863872f3e004b9e94d579c3f45da060]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:40.781504 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e76c7f6ac6a477ba2af13b95c12858f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.781546 11298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e76c7f6ac6a477ba2af13b95c12858f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.786880 11609 tablet_bootstrap.cc:492] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Bootstrap starting.
I20260928 15:02:40.787621 11438 tablet_service.cc:1467] Processing CreateTablet for tablet 1e76c7f6ac6a477ba2af13b95c12858f (DEFAULT_TABLE table=test-table [id=2863872f3e004b9e94d579c3f45da060]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:40.788074 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e76c7f6ac6a477ba2af13b95c12858f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.788686 11609 tablet_bootstrap.cc:654] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.793350 11610 tablet_bootstrap.cc:492] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Bootstrap starting.
I20260928 15:02:40.794679 11610 tablet_bootstrap.cc:654] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.796665 11613 tablet_bootstrap.cc:492] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Bootstrap starting.
I20260928 15:02:40.796974 11609 tablet_bootstrap.cc:492] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: No bootstrap required, opened a new log
I20260928 15:02:40.797133 11609 ts_tablet_manager.cc:1397] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 15:02:40.797744 11609 raft_consensus.cc:348] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.797950 11609 raft_consensus.cc:374] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.798059 11609 raft_consensus.cc:729] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cda83718c3d41efb8ac3335f015f864, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.798120 11613 tablet_bootstrap.cc:654] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.798357 11609 consensus_queue.cc:260] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.799182 11609 ts_tablet_manager.cc:1428] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:02:40.800089 11610 tablet_bootstrap.cc:492] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: No bootstrap required, opened a new log
I20260928 15:02:40.800225 11613 tablet_bootstrap.cc:492] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: No bootstrap required, opened a new log
I20260928 15:02:40.800225 11610 ts_tablet_manager.cc:1397] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 15:02:40.800400 11613 ts_tablet_manager.cc:1397] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260928 15:02:40.801034 11610 raft_consensus.cc:348] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.801278 11610 raft_consensus.cc:374] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.801345 11610 raft_consensus.cc:729] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275431e5965d46d78c5ce963060b1c27, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.801267 11613 raft_consensus.cc:348] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.801493 11613 raft_consensus.cc:374] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.801589 11613 raft_consensus.cc:729] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.801565 11610 consensus_queue.cc:260] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.801818 11613 consensus_queue.cc:260] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.802376 11610 ts_tablet_manager.cc:1428] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:40.802537 11613 ts_tablet_manager.cc:1428] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:40.987942 11586 raft_consensus.cc:482] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:40.988173 11586 raft_consensus.cc:504] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.988865 11586 leader_election.cc:288] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:40.989574 11322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e76c7f6ac6a477ba2af13b95c12858f" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27" is_pre_election: true
I20260928 15:02:40.989583 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e76c7f6ac6a477ba2af13b95c12858f" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864" is_pre_election: true
I20260928 15:02:40.989835 11322 raft_consensus.cc:2454] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:40.989835 11184 raft_consensus.cc:2454] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 0.
I20260928 15:02:40.990361 11391 leader_election.cc:302] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:40.990640 11586 raft_consensus.cc:2781] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:40.990768 11586 raft_consensus.cc:482] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:40.990876 11586 raft_consensus.cc:3037] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.993239 11586 raft_consensus.cc:504] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:40.993777 11586 leader_election.cc:288] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Requested vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233)
I20260928 15:02:40.994290 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e76c7f6ac6a477ba2af13b95c12858f" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864"
I20260928 15:02:40.994336 11322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e76c7f6ac6a477ba2af13b95c12858f" candidate_uuid: "3aa6beb4f44248c0af4effcfbf21499b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275431e5965d46d78c5ce963060b1c27"
I20260928 15:02:40.994531 11322 raft_consensus.cc:3037] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.994531 11184 raft_consensus.cc:3037] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.997485 11184 raft_consensus.cc:2454] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:40.997485 11322 raft_consensus.cc:2454] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa6beb4f44248c0af4effcfbf21499b in term 1.
I20260928 15:02:40.997885 11391 leader_election.cc:302] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cda83718c3d41efb8ac3335f015f864, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:40.998131 11586 raft_consensus.cc:2781] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:40.998283 11586 raft_consensus.cc:686] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Becoming Leader. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Running, Role: LEADER
I20260928 15:02:40.998539 11586 consensus_queue.cc:237] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.001220 11024 catalog_manager.cc:5495] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195). New cstate: current_term: 1 leader_uuid: "3aa6beb4f44248c0af4effcfbf21499b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:41.046602 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.039527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.9.12.195:43405)
0928 15:02:41.040439 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.040445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.040477 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:02:41.040539 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.040543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.040614 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.040699 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:41.041305 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.042287 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.044657 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.045084 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.045186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.045823 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:41.045935 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:41.046003 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.046175 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.046238 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":555,"threads_started":1}
I20260928 15:02:41.056609 11322 raft_consensus.cc:1261] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:41.056779 11184 raft_consensus.cc:1261] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Refusing update from remote peer 3aa6beb4f44248c0af4effcfbf21499b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:41.057256 11534 consensus_queue.cc:1035] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:41.057600 11586 consensus_queue.cc:1035] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:41.061770 11582 mvcc.cc:205] Tried to move back new op lower bound from 7334329389282615296 to 7334329389051789312. Current Snapshot: MvccSnapshot[applied={T|T < 7334329389282615296}]
I20260928 15:02:41.107012 11023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:41.107317 11023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:41.110646 11023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Sending DeleteTablet for 3 replicas of tablet 1e76c7f6ac6a477ba2af13b95c12858f
I20260928 15:02:41.111464 11160 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e76c7f6ac6a477ba2af13b95c12858f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:41 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:35108
I20260928 15:02:41.111662 11438 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e76c7f6ac6a477ba2af13b95c12858f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:41 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:41560
I20260928 15:02:41.111804 11598 tablet_replica.cc:330] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: stopping tablet replica
I20260928 15:02:41.111967 11600 tablet_replica.cc:330] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: stopping tablet replica
I20260928 15:02:41.112022 11598 raft_consensus.cc:2229] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:41.112210 11598 raft_consensus.cc:2258] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:41.112210 11600 raft_consensus.cc:2229] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:41.112543 11600 raft_consensus.cc:2258] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:41.113273 11298 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e76c7f6ac6a477ba2af13b95c12858f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:41 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:57184
I20260928 15:02:41.113541 11599 tablet_replica.cc:330] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: stopping tablet replica
I20260928 15:02:41.113662 11600 ts_tablet_manager.cc:1905] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:41.113735 11599 raft_consensus.cc:2229] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:41.113555 11023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:02:41.113903 11599 raft_consensus.cc:2258] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 15:02:41.114250 11023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:41.115087 11598 ts_tablet_manager.cc:1905] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:41.120252 11600 ts_tablet_manager.cc:1918] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:41.120493 11600 log.cc:1196] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/1e76c7f6ac6a477ba2af13b95c12858f
I20260928 15:02:41.120934 11600 ts_tablet_manager.cc:1939] T 1e76c7f6ac6a477ba2af13b95c12858f P 3aa6beb4f44248c0af4effcfbf21499b: Deleting consensus metadata
I20260928 15:02:41.121799 11598 ts_tablet_manager.cc:1918] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:41.121945 11598 log.cc:1196] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/1e76c7f6ac6a477ba2af13b95c12858f
I20260928 15:02:41.122244 11010 catalog_manager.cc:4841] TS 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405): tablet 1e76c7f6ac6a477ba2af13b95c12858f (table test-table [id=2863872f3e004b9e94d579c3f45da060]) successfully deleted
I20260928 15:02:41.122327 11598 ts_tablet_manager.cc:1939] T 1e76c7f6ac6a477ba2af13b95c12858f P 2cda83718c3d41efb8ac3335f015f864: Deleting consensus metadata
I20260928 15:02:41.123656 11010 catalog_manager.cc:4841] TS 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635): tablet 1e76c7f6ac6a477ba2af13b95c12858f (table test-table [id=2863872f3e004b9e94d579c3f45da060]) successfully deleted
I20260928 15:02:41.123973 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 012b681d26cb4a5281fed5e5c1daee4f (DEFAULT_TABLE table=test-table [id=6e64eeb019ff4a9eabc3ae4ec81b2193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:41.124204 11298 tablet_service.cc:1467] Processing CreateTablet for tablet 012b681d26cb4a5281fed5e5c1daee4f (DEFAULT_TABLE table=test-table [id=6e64eeb019ff4a9eabc3ae4ec81b2193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:41.124411 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 012b681d26cb4a5281fed5e5c1daee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:41.124738 11298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 012b681d26cb4a5281fed5e5c1daee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:41.125025 11599 ts_tablet_manager.cc:1905] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:41.128125 11438 tablet_service.cc:1467] Processing CreateTablet for tablet 012b681d26cb4a5281fed5e5c1daee4f (DEFAULT_TABLE table=test-table [id=6e64eeb019ff4a9eabc3ae4ec81b2193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:41.128593 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 012b681d26cb4a5281fed5e5c1daee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:41.131712 11599 ts_tablet_manager.cc:1918] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:41.132220 11599 log.cc:1196] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/1e76c7f6ac6a477ba2af13b95c12858f
I20260928 15:02:41.132683 11599 ts_tablet_manager.cc:1939] T 1e76c7f6ac6a477ba2af13b95c12858f P 275431e5965d46d78c5ce963060b1c27: Deleting consensus metadata
I20260928 15:02:41.133772 11010 catalog_manager.cc:4841] TS 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233): tablet 1e76c7f6ac6a477ba2af13b95c12858f (table test-table [id=2863872f3e004b9e94d579c3f45da060]) successfully deleted
I20260928 15:02:41.135284 11610 tablet_bootstrap.cc:492] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Bootstrap starting.
I20260928 15:02:41.136057 11609 tablet_bootstrap.cc:492] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Bootstrap starting.
I20260928 15:02:41.136884 11610 tablet_bootstrap.cc:654] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:41.137712 11609 tablet_bootstrap.cc:654] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:41.137962 11613 tablet_bootstrap.cc:492] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Bootstrap starting.
I20260928 15:02:41.139698 11613 tablet_bootstrap.cc:654] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:41.145818 11610 tablet_bootstrap.cc:492] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: No bootstrap required, opened a new log
I20260928 15:02:41.146036 11610 ts_tablet_manager.cc:1397] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20260928 15:02:41.146663 11609 tablet_bootstrap.cc:492] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: No bootstrap required, opened a new log
I20260928 15:02:41.146862 11609 ts_tablet_manager.cc:1397] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260928 15:02:41.146873 11610 raft_consensus.cc:348] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.147185 11613 tablet_bootstrap.cc:492] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: No bootstrap required, opened a new log
I20260928 15:02:41.147253 11610 raft_consensus.cc:374] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:41.147396 11610 raft_consensus.cc:729] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275431e5965d46d78c5ce963060b1c27, State: Initialized, Role: FOLLOWER
I20260928 15:02:41.147435 11613 ts_tablet_manager.cc:1397] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260928 15:02:41.147644 11609 raft_consensus.cc:348] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.147728 11610 consensus_queue.cc:260] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.147915 11609 raft_consensus.cc:374] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:41.148010 11609 raft_consensus.cc:729] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cda83718c3d41efb8ac3335f015f864, State: Initialized, Role: FOLLOWER
I20260928 15:02:41.148434 11613 raft_consensus.cc:348] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.148751 11613 raft_consensus.cc:374] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:41.148857 11613 raft_consensus.cc:729] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa6beb4f44248c0af4effcfbf21499b, State: Initialized, Role: FOLLOWER
I20260928 15:02:41.149183 11613 consensus_queue.cc:260] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.149298 11609 consensus_queue.cc:260] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.150038 11613 ts_tablet_manager.cc:1428] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:41.151409 11609 ts_tablet_manager.cc:1428] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 15:02:41.152060 11610 ts_tablet_manager.cc:1428] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260928 15:02:41.220196 11616 raft_consensus.cc:482] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:41.220470 11616 raft_consensus.cc:504] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.222357 11616 leader_election.cc:288] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405)
I20260928 15:02:41.247851 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.222742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37635 (local address 127.9.12.194:56521)
0928 15:02:41.237554 (+ 14812us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:41.237583 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:41.237732 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:41.241206 (+ 3474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:41.241213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:41.241250 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:41.241620 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.241633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.243326 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.243334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.245293 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.245308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.247002 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.247008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.247181 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.247218 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:02:41.247286 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:02:41.247419 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14663,"mutex_wait_us":46,"thread_start_us":14512,"threads_started":1}
I20260928 15:02:41.249089 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.222827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:56521 (local address 127.9.12.193:37635)
0928 15:02:41.240730 (+ 17903us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.240736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.240766 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:02:41.240916 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.240923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.241004 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.241774 (+ 770us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:41.241981 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.243167 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.245491 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.246756 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.248454 (+ 1698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.248538 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.248598 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.248726 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17771,"thread_start_us":17626,"threads_started":1}
I20260928 15:02:41.249277 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.238075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:58315 (local address 127.9.12.195:43405)
0928 15:02:41.238292 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.238301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.241230 (+ 2929us) server_negotiation.cc:407] Connection header received
0928 15:02:41.241495 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.241505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.241617 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.241692 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:41.242595 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.243652 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.246656 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.247985 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.248134 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.248204 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.248883 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.249010 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:02:41.249805 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "012b681d26cb4a5281fed5e5c1daee4f" candidate_uuid: "275431e5965d46d78c5ce963060b1c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864" is_pre_election: true
I20260928 15:02:41.250068 11185 raft_consensus.cc:2454] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 275431e5965d46d78c5ce963060b1c27 in term 0.
I20260928 15:02:41.250226 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.238474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43405 (local address 127.9.12.194:58315)
0928 15:02:41.241122 (+ 2648us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:41.241144 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:41.241325 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:41.241978 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:41.241982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:41.242012 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:41.242395 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.242407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.243826 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.243832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.246487 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.246502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.248534 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.248540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.248702 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.248731 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:02:41.249625 (+ 894us) client_negotiation.cc:240] Negotiation successful
0928 15:02:41.249748 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2509,"thread_start_us":3802,"threads_started":1}
I20260928 15:02:41.250664 11252 leader_election.cc:302] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 2cda83718c3d41efb8ac3335f015f864; no voters:
I20260928 15:02:41.251155 11616 raft_consensus.cc:2781] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:41.251307 11616 raft_consensus.cc:482] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:41.251399 11616 raft_consensus.cc:3037] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:41.251955 11458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "012b681d26cb4a5281fed5e5c1daee4f" candidate_uuid: "275431e5965d46d78c5ce963060b1c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa6beb4f44248c0af4effcfbf21499b" is_pre_election: true
I20260928 15:02:41.252467 11458 raft_consensus.cc:2454] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 275431e5965d46d78c5ce963060b1c27 in term 0.
I20260928 15:02:41.254438 11616 raft_consensus.cc:504] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.255074 11616 leader_election.cc:288] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [CANDIDATE]: Term 1 election: Requested vote from peers 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635), 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405)
I20260928 15:02:41.255475 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "012b681d26cb4a5281fed5e5c1daee4f" candidate_uuid: "275431e5965d46d78c5ce963060b1c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cda83718c3d41efb8ac3335f015f864"
I20260928 15:02:41.255733 11185 raft_consensus.cc:3037] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:41.255793 11458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "012b681d26cb4a5281fed5e5c1daee4f" candidate_uuid: "275431e5965d46d78c5ce963060b1c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa6beb4f44248c0af4effcfbf21499b"
I20260928 15:02:41.255991 11458 raft_consensus.cc:3037] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:41.259037 11185 raft_consensus.cc:2454] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 275431e5965d46d78c5ce963060b1c27 in term 1.
I20260928 15:02:41.259198 11458 raft_consensus.cc:2454] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 275431e5965d46d78c5ce963060b1c27 in term 1.
I20260928 15:02:41.259639 11252 leader_election.cc:302] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275431e5965d46d78c5ce963060b1c27, 3aa6beb4f44248c0af4effcfbf21499b; no voters:
I20260928 15:02:41.259907 11616 raft_consensus.cc:2781] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:41.260473 11616 raft_consensus.cc:686] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 1 LEADER]: Becoming Leader. State: Replica: 275431e5965d46d78c5ce963060b1c27, State: Running, Role: LEADER
I20260928 15:02:41.261061 11616 consensus_queue.cc:237] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } }
I20260928 15:02:41.264614 11022 catalog_manager.cc:5495] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 275431e5965d46d78c5ce963060b1c27 (127.9.12.194). New cstate: current_term: 1 leader_uuid: "275431e5965d46d78c5ce963060b1c27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275431e5965d46d78c5ce963060b1c27" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:41.323737 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.314552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.9.12.194:36233)
0928 15:02:41.315564 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.315572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.315599 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:41.315711 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.315718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.315835 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.316054 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:41.316656 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.318079 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.320363 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.321161 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.321327 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.321960 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:41.322893 (+ 933us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:41.323069 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.323185 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.323289 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":719,"threads_started":1}
I20260928 15:02:41.345276 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.336777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:60047 (local address 127.9.12.195:43405)
0928 15:02:41.336984 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.336992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.337712 (+ 720us) server_negotiation.cc:407] Connection header received
0928 15:02:41.337919 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.337928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.338044 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.338148 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:41.338795 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.340174 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.342724 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.343564 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.343716 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.344356 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:41.344513 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:41.344610 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.344764 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.344872 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:02:41.345621 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.336782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43405 (local address 127.9.12.194:60047)
0928 15:02:41.337582 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:41.337608 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:41.337771 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:41.338189 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:41.338197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:41.338213 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:41.338619 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.338633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.340335 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.340342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.342562 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.342581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.343776 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.343784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.344041 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.344128 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:41.344621 (+ 493us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:41.344635 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:02:41.344728 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:02:41.344904 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":426,"threads_started":1}
I20260928 15:02:41.358768 11185 raft_consensus.cc:1261] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Refusing update from remote peer 275431e5965d46d78c5ce963060b1c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:41.359040 11458 raft_consensus.cc:1261] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Refusing update from remote peer 275431e5965d46d78c5ce963060b1c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:41.359692 11631 consensus_queue.cc:1035] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cda83718c3d41efb8ac3335f015f864" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:41.360116 11616 consensus_queue.cc:1035] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa6beb4f44248c0af4effcfbf21499b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:41.417951 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.405645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36233 (local address 127.9.12.195:39165)
0928 15:02:41.411570 (+ 5925us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:41.411599 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:41.411761 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:41.412102 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:41.412109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:41.412127 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:41.412589 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.412605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.413647 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.413655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.415823 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:41.415844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.416720 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.416730 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:41.416946 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.417017 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:41.417475 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:41.417491 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:02:41.417576 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:02:41.417678 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5721,"thread_start_us":5539,"threads_started":1}
I20260928 15:02:41.418100 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.405955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:39165 (local address 127.9.12.194:36233)
0928 15:02:41.406168 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.406178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.411702 (+ 5524us) server_negotiation.cc:407] Connection header received
0928 15:02:41.411906 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.411910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.411977 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.412053 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:41.412770 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.413489 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.415987 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.416534 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.416676 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.417190 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:41.417344 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:41.417441 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.417617 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.417718 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 15:02:41.419229 11582 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:02:43.032140 11021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:43.032431 11021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:43.036401 11021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cf097a382d74937854d1e54914bd258: Sending DeleteTablet for 3 replicas of tablet 012b681d26cb4a5281fed5e5c1daee4f
I20260928 15:02:43.037137 11160 tablet_service.cc:1514] Processing DeleteTablet for tablet 012b681d26cb4a5281fed5e5c1daee4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:43 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:35108
I20260928 15:02:43.037243 11298 tablet_service.cc:1514] Processing DeleteTablet for tablet 012b681d26cb4a5281fed5e5c1daee4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:43 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:57184
I20260928 15:02:43.037358 11438 tablet_service.cc:1514] Processing DeleteTablet for tablet 012b681d26cb4a5281fed5e5c1daee4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:43 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:41560
I20260928 15:02:43.037737 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11095
I20260928 15:02:43.049705 11666 tablet_replica.cc:330] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: stopping tablet replica
I20260928 15:02:43.049727 11665 tablet_replica.cc:330] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: stopping tablet replica
I20260928 15:02:43.049937 11665 raft_consensus.cc:2229] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:43.050406 11665 raft_consensus.cc:2258] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:43.050520 11666 raft_consensus.cc:2229] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:43.051718 11665 ts_tablet_manager.cc:1905] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:43.052012 11666 raft_consensus.cc:2258] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:43.053346 11666 ts_tablet_manager.cc:1905] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:43.056810 11667 tablet_replica.cc:330] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: stopping tablet replica
I20260928 15:02:43.057083 11667 raft_consensus.cc:2229] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:43.057410 11667 raft_consensus.cc:2258] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:43.058622 11667 ts_tablet_manager.cc:1905] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:43.059453 11665 ts_tablet_manager.cc:1918] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:43.059652 11665 log.cc:1196] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/012b681d26cb4a5281fed5e5c1daee4f
I20260928 15:02:43.060194 11665 ts_tablet_manager.cc:1939] T 012b681d26cb4a5281fed5e5c1daee4f P 2cda83718c3d41efb8ac3335f015f864: Deleting consensus metadata
I20260928 15:02:43.060727 11666 ts_tablet_manager.cc:1918] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:43.060937 11666 log.cc:1196] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/012b681d26cb4a5281fed5e5c1daee4f
I20260928 15:02:43.061477 11666 ts_tablet_manager.cc:1939] T 012b681d26cb4a5281fed5e5c1daee4f P 275431e5965d46d78c5ce963060b1c27: Deleting consensus metadata
I20260928 15:02:43.061510 11010 catalog_manager.cc:4841] TS 2cda83718c3d41efb8ac3335f015f864 (127.9.12.193:37635): tablet 012b681d26cb4a5281fed5e5c1daee4f (table test-table [id=6e64eeb019ff4a9eabc3ae4ec81b2193]) successfully deleted
I20260928 15:02:43.062698 11010 catalog_manager.cc:4841] TS 275431e5965d46d78c5ce963060b1c27 (127.9.12.194:36233): tablet 012b681d26cb4a5281fed5e5c1daee4f (table test-table [id=6e64eeb019ff4a9eabc3ae4ec81b2193]) successfully deleted
I20260928 15:02:43.065366 11667 ts_tablet_manager.cc:1918] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:43.065511 11667 log.cc:1196] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/012b681d26cb4a5281fed5e5c1daee4f
I20260928 15:02:43.065984 11667 ts_tablet_manager.cc:1939] T 012b681d26cb4a5281fed5e5c1daee4f P 3aa6beb4f44248c0af4effcfbf21499b: Deleting consensus metadata
Sep 28 15:02:43 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:43.067116 11010 catalog_manager.cc:4841] TS 3aa6beb4f44248c0af4effcfbf21499b (127.9.12.195:43405): tablet 012b681d26cb4a5281fed5e5c1daee4f (table test-table [id=6e64eeb019ff4a9eabc3ae4ec81b2193]) successfully deleted
I20260928 15:02:43.073139 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:43.038896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.9.12.193:37635)
0928 15:02:43.055925 (+ 17029us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:43.055929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:43.055949 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:02:43.055995 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:43.055999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:43.056053 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:43.056182 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:43.058882 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:43.060098 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:43.061955 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:43.062597 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:43.062738 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:43.067832 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:43.067845 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:43.067861 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:43.067893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:43.070633 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:43.071219 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:43.071226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:43.071228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:43.071391 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:43.071710 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:43.071713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:43.071715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:43.072121 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:43.072420 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:43.072644 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 15:02:43.072803 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16909,"thread_start_us":17428,"threads_started":1}
W20260928 15:02:43.073885 11225 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35124
W20260928 15:02:43.074506 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:43.074728 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 11095
I20260928 15:02:43.094142 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11234
Sep 28 15:02:43 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:43.109458 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:43.094919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.9.12.194:36233)
0928 15:02:43.095482 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:43.095486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:43.095501 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:43.095554 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:43.095557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:43.095618 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:43.095749 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:43.097157 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:43.097918 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:43.099212 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:43.099641 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:43.099772 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:43.103959 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:43.103973 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:43.103988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:43.104019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:43.106659 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:43.107345 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:43.107352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:43.107354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:43.107481 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:43.107845 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:43.107850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:43.107853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:43.108204 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:43.108558 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:43.108881 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 15:02:43.109113 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":309,"threads_started":1}
W20260928 15:02:43.110150 11364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57200
W20260928 15:02:43.110739 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11234: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:43.110863 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 11234
I20260928 15:02:43.133180 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11373
Sep 28 15:02:43 dist-test-slave-dmhs krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:43.148450 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:43.134044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.9.12.195:43405)
0928 15:02:43.134519 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:43.134524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:43.134540 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:02:43.134594 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:43.134598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:43.134650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:43.134745 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:43.135966 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:43.136825 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:43.138044 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:43.138424 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:43.138513 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:43.142765 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:43.142779 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:43.142795 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:43.142827 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:43.145637 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:43.146309 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:43.146317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:43.146320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:43.146439 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:43.146825 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:43.146832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:43.146835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:43.147203 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:43.147522 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:43.147793 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 15:02:43.148023 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
W20260928 15:02:43.149169 11503 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41602
W20260928 15:02:43.149724 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11373: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:43.149916 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 11373
I20260928 15:02:43.176018 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 10992
I20260928 15:02:43.177372 11064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:43.296124 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 10992
2026-09-28T15:02:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8369 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:43 dist-test-slave-dmhs krb5kdc[11675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:43 dist-test-slave-dmhs krb5kdc[11675](info): set up 2 sockets
Sep 28 15:02:43 dist-test-slave-dmhs krb5kdc[11675](info): commencing operation
krb5kdc: starting...
W20260928 15:02:44.364916 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:44 dist-test-slave-dmhs krb5kdc[11675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:44.562083 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37737
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:37737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:44.880729 11688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:44.881012 11688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:44.881186 11688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:44.888370 11688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:44.888477 11688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:44.888568 11688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:44.888670 11688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:44.888728 11688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:44.898362 11688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:37737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:44.899767 11688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:44.901156 11688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:44.912814 11693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:44.912767 11696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:44.912838 11694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:44.914109 11688 server_base.cc:1034] running on GCE node
I20260928 15:02:44.914848 11688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:44.916190 11688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:44.917430 11688 hybrid_clock.cc:648] HybridClock initialized: now 1790607764917381 us; error 38 us; skew 500 ppm
I20260928 15:02:44.917807 11688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:44 dist-test-slave-dmhs krb5kdc[11675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607764, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:44.921031 11688 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:44.922376 11688 webserver.cc:460] Webserver started at http://127.9.12.254:42267/ using document root <none> and password file <none>
I20260928 15:02:44.923007 11688 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:44.923098 11688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:44.923349 11688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:44.926200 11688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "0c6cab55c921414cb8b2aae97bba5b0e"
format_stamp: "Formatted at 2026-09-28 15:02:44 on dist-test-slave-dmhs"
server_key: "4ac3b922df60e1a7f9f8587127236b16"
server_key_iv: "0f245239795d958c745973207fffa37a"
server_key_version: "encryptionkey@0"
I20260928 15:02:44.926949 11688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "0c6cab55c921414cb8b2aae97bba5b0e"
format_stamp: "Formatted at 2026-09-28 15:02:44 on dist-test-slave-dmhs"
server_key: "4ac3b922df60e1a7f9f8587127236b16"
server_key_iv: "0f245239795d958c745973207fffa37a"
server_key_version: "encryptionkey@0"
I20260928 15:02:44.931820 11688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:02:44.935082 11703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:44.936401 11688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:44.936591 11688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "0c6cab55c921414cb8b2aae97bba5b0e"
format_stamp: "Formatted at 2026-09-28 15:02:44 on dist-test-slave-dmhs"
server_key: "4ac3b922df60e1a7f9f8587127236b16"
server_key_iv: "0f245239795d958c745973207fffa37a"
server_key_version: "encryptionkey@0"
I20260928 15:02:44.936774 11688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:44.955438 11688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:44.959369 11688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:44.959676 11688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:44.977964 11688 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:37737
I20260928 15:02:44.977998 11754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:37737 every 8 connection(s)
I20260928 15:02:44.979360 11688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:44.980243 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 11688
I20260928 15:02:44.980533 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:44.981220 9267 external_mini_cluster.cc:1428] Setting key 60e99308f54acb8dd3d2725b0d09413c
I20260928 15:02:44.983868 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:44.994576 11755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e: Bootstrap starting.
Sep 28 15:02:44 dist-test-slave-dmhs krb5kdc[11675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:44.998203 11755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:44.999272 11755 log.cc:824] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:45.002497 11755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e: No bootstrap required, opened a new log
I20260928 15:02:45.003232 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:44.983188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.9.12.254:37737)
0928 15:02:44.983957 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:44.983973 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:44.984028 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:02:44.985727 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:44.985750 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:44.986125 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:44.986585 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:44.988101 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:44.989490 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:44.990806 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:44.991493 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:44.991723 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:44.996026 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:44.996033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:44.996046 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:44.996091 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:44.999088 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:44.999891 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:44.999901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:44.999912 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.000136 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.000551 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.000559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.000573 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.001083 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:45.001286 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:45.001771 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 15:02:45.002094 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":335,"threads_started":1}
I20260928 15:02:45.010550 11755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37737 } }
I20260928 15:02:45.010913 11755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:45.011037 11755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6cab55c921414cb8b2aae97bba5b0e, State: Initialized, Role: FOLLOWER
I20260928 15:02:45.011581 11755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37737 } }
I20260928 15:02:45.011801 11755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:45.011933 11755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:45.012102 11755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:45.014155 11755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37737 } }
I20260928 15:02:45.014613 11755 leader_election.cc:302] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6cab55c921414cb8b2aae97bba5b0e; no voters:
I20260928 15:02:45.015149 11755 leader_election.cc:288] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:45.015280 11760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:45.015934 11760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [term 1 LEADER]: Becoming Leader. State: Replica: 0c6cab55c921414cb8b2aae97bba5b0e, State: Running, Role: LEADER
I20260928 15:02:45.016451 11760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37737 } }
I20260928 15:02:45.017167 11755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:45.019596 11762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6cab55c921414cb8b2aae97bba5b0e. Latest consensus state: current_term: 1 leader_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37737 } } }
I20260928 15:02:45.019829 11761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6cab55c921414cb8b2aae97bba5b0e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37737 } } }
I20260928 15:02:45.020184 11762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:45.020341 11761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:45.024183 11768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:45.030260 11768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:45.038950 11768 catalog_manager.cc:1347] Generated new cluster ID: 7f421613913a4edea48d7d1e96541ff2
I20260928 15:02:45.039100 11768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:45.050442 11768 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:45.051782 11768 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:45.059613 11768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c6cab55c921414cb8b2aae97bba5b0e: Generated new TSK 0
I20260928 15:02:45.060513 11768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:45.107586 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37737
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:45.431322 11781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:45.431625 11781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:45.431790 11781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:45.439430 11781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:45.439549 11781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:45.439759 11781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:45.450160 11781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:45.451472 11781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:45.452943 11781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:45.466575 11786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.466574 11787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.467072 11789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.467653 11781 server_base.cc:1034] running on GCE node
I20260928 15:02:45.468389 11781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:45.469378 11781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:45.470613 11781 hybrid_clock.cc:648] HybridClock initialized: now 1790607765470552 us; error 54 us; skew 500 ppm
I20260928 15:02:45.470942 11781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:45 dist-test-slave-dmhs krb5kdc[11675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:45.474249 11781 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:45.476012 11781 webserver.cc:460] Webserver started at http://127.9.12.193:40563/ using document root <none> and password file <none>
I20260928 15:02:45.476826 11781 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:45.476981 11781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:45.477294 11781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:45.480429 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "d00ee88f473946cb9dab139bfe4ffe14"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-dmhs"
server_key: "9a37fb7619d5a0ef5393d53c09e9da36"
server_key_iv: "5cd898ec32cd969b99b6e2aa27cd1d63"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.481307 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "d00ee88f473946cb9dab139bfe4ffe14"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-dmhs"
server_key: "9a37fb7619d5a0ef5393d53c09e9da36"
server_key_iv: "5cd898ec32cd969b99b6e2aa27cd1d63"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.487376 11781 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 15:02:45.491245 11796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.492892 11781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:45.493100 11781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "d00ee88f473946cb9dab139bfe4ffe14"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-dmhs"
server_key: "9a37fb7619d5a0ef5393d53c09e9da36"
server_key_iv: "5cd898ec32cd969b99b6e2aa27cd1d63"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.493333 11781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:45.517643 11781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:45.522243 11781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:45.522557 11781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:45.523614 11781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:45.525014 11781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:45.525118 11781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.525269 11781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:45.525342 11781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.559667 11781 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:37627
I20260928 15:02:45.559731 11909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:37627 every 8 connection(s)
I20260928 15:02:45.561053 11781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:45.567553 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 11781
I20260928 15:02:45.567790 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:45.568651 9267 external_mini_cluster.cc:1428] Setting key b01dd15c33ff8ac579b9ff1623c3f01c
Sep 28 15:02:45 dist-test-slave-dmhs krb5kdc[11675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:02:45.589231 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.563644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42119 (local address 127.9.12.254:37737)
0928 15:02:45.564116 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:45.564122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:45.565197 (+ 1075us) server_negotiation.cc:407] Connection header received
0928 15:02:45.567095 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:45.567103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:45.567203 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:45.567303 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:45.569606 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.570636 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.573303 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.573703 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.573811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.581887 (+ 8076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:45.581896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:45.581904 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:45.581955 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:45.585730 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.586562 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.586569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.586575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.586734 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.587173 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.587181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.587186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.587598 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:45.587756 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:45.588539 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 15:02:45.588797 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":196,"threads_started":1}
I20260928 15:02:45.590153 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.564224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37737 (local address 127.9.12.193:42119)
0928 15:02:45.565038 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:45.565077 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:45.566757 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:45.567623 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:45.567637 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:45.568528 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:45.569404 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:45.569425 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.570857 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.570864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:45.573126 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:45.573145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.573960 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.573970 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:45.574309 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.579081 (+ 4772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:45.579108 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:45.581585 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:45.585916 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:45.585923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:45.585946 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:45.586399 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:45.586844 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:45.586847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:45.586849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:45.587043 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:45.587728 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:45.587734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:45.588080 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 15:02:45.588378 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 15:02:45.588863 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":285,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:02:45.592044 11910 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37737
I20260928 15:02:45.592411 11910 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:45.593128 11910 heartbeater.cc:511] Master 127.9.12.254:37737 requested a full tablet report, sending...
I20260928 15:02:45.595243 11720 ts_manager.cc:194] Registered new tserver with Master: d00ee88f473946cb9dab139bfe4ffe14 (127.9.12.193:37627)
I20260928 15:02:45.596779 11720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:42119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:45.616401 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37737
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:45.941207 11916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:45.941576 11916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:45.941752 11916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:45.948938 11916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:45.949092 11916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:45.949290 11916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:45.958973 11916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:45.960282 11916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:45.961735 11916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:45.974625 11921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.974660 11922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.974720 11924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.975803 11916 server_base.cc:1034] running on GCE node
I20260928 15:02:45.976527 11916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:45.977604 11916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:45.978825 11916 hybrid_clock.cc:648] HybridClock initialized: now 1790607765978770 us; error 48 us; skew 500 ppm
I20260928 15:02:45.979159 11916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:45 dist-test-slave-dmhs krb5kdc[11675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:45.982324 11916 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:45.983891 11916 webserver.cc:460] Webserver started at http://127.9.12.194:43001/ using document root <none> and password file <none>
I20260928 15:02:45.984649 11916 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:45.984772 11916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:45.985100 11916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:45.988219 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "af15c3009120431e8e54e5213df5c8b7"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-dmhs"
server_key: "dd91d0d7bba02cdcf4b0fa0bff1b3e1a"
server_key_iv: "f40d2351f2bdf0ab21c0ec80e1aa4191"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.989161 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "af15c3009120431e8e54e5213df5c8b7"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-dmhs"
server_key: "dd91d0d7bba02cdcf4b0fa0bff1b3e1a"
server_key_iv: "f40d2351f2bdf0ab21c0ec80e1aa4191"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.994844 11916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:02:45.999455 11931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.001952 11916 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 15:02:46.002112 11916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "af15c3009120431e8e54e5213df5c8b7"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-dmhs"
server_key: "dd91d0d7bba02cdcf4b0fa0bff1b3e1a"
server_key_iv: "f40d2351f2bdf0ab21c0ec80e1aa4191"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.002285 11916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:46.017757 11916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:46.025493 11916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:46.025821 11916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:46.026942 11916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:46.028406 11916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:46.028486 11916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.028606 11916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:46.028688 11916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.062623 11916 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:43355
I20260928 15:02:46.062680 12043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:43355 every 8 connection(s)
I20260928 15:02:46.064003 11916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:46.065726 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 11916
I20260928 15:02:46.065930 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:46.066445 9267 external_mini_cluster.cc:1428] Setting key f7bbfafd918a06f6de9ad021d5311430
Sep 28 15:02:46 dist-test-slave-dmhs krb5kdc[11675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:02:46.088235 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:46.066972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:51265 (local address 127.9.12.254:37737)
0928 15:02:46.067148 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:46.067153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:46.068631 (+ 1478us) server_negotiation.cc:407] Connection header received
0928 15:02:46.071434 (+ 2803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:46.071439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:46.071508 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:46.071603 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:46.073742 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.074485 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.076188 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.077044 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.077185 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:46.080776 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:46.080785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:46.080793 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:46.080839 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:46.084620 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:46.085382 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:46.085390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:46.085394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:46.085578 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:46.086400 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:46.086405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:46.086409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:46.086802 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:46.087002 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:46.087802 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 15:02:46.087980 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:02:46.089293 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:46.067632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37737 (local address 127.9.12.194:51265)
0928 15:02:46.068477 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:46.068523 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:46.071136 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:46.071881 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:46.071894 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:46.072728 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:46.073545 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:46.073573 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.074654 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.074662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:46.076037 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:46.076048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.077244 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.077249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:46.077441 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:46.078141 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:46.078162 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:46.080550 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:46.084766 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:46.084773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:46.084785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:46.085220 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:46.085817 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:46.085823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:46.085826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:46.086165 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:46.086978 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:46.087006 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:46.087351 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 15:02:46.087674 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 15:02:46.088088 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":341,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
I20260928 15:02:46.090904 12044 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37737
I20260928 15:02:46.091267 12044 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:46.092028 12044 heartbeater.cc:511] Master 127.9.12.254:37737 requested a full tablet report, sending...
I20260928 15:02:46.093676 11720 ts_manager.cc:194] Registered new tserver with Master: af15c3009120431e8e54e5213df5c8b7 (127.9.12.194:43355)
I20260928 15:02:46.094898 11720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:51265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:46.116559 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37737
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:46.430825 12049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:46.431149 12049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:46.431283 12049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:46.438503 12049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:46.438653 12049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:46.438834 12049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:46.450881 12049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:46.452653 12049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:46.454010 12049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:46.467175 12054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:46.467245 12055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:46.467211 12057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:46.467876 12049 server_base.cc:1034] running on GCE node
I20260928 15:02:46.468575 12049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:46.469477 12049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:46.470702 12049 hybrid_clock.cc:648] HybridClock initialized: now 1790607766470652 us; error 44 us; skew 500 ppm
I20260928 15:02:46.471019 12049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:46 dist-test-slave-dmhs krb5kdc[11675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:46.474291 12049 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:46.475744 12049 webserver.cc:460] Webserver started at http://127.9.12.195:45143/ using document root <none> and password file <none>
I20260928 15:02:46.476495 12049 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:46.476630 12049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:46.476946 12049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:46.479800 12049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "778041d62c214b8da3b8447b4629085e"
format_stamp: "Formatted at 2026-09-28 15:02:46 on dist-test-slave-dmhs"
server_key: "0b13deb146fe48dc6f12d27297adc5d0"
server_key_iv: "a08d2b4da9b35f7e8dc3f05ab3192345"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.480657 12049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "778041d62c214b8da3b8447b4629085e"
format_stamp: "Formatted at 2026-09-28 15:02:46 on dist-test-slave-dmhs"
server_key: "0b13deb146fe48dc6f12d27297adc5d0"
server_key_iv: "a08d2b4da9b35f7e8dc3f05ab3192345"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.485869 12049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 15:02:46.489941 12064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.491289 12049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:02:46.491497 12049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "778041d62c214b8da3b8447b4629085e"
format_stamp: "Formatted at 2026-09-28 15:02:46 on dist-test-slave-dmhs"
server_key: "0b13deb146fe48dc6f12d27297adc5d0"
server_key_iv: "a08d2b4da9b35f7e8dc3f05ab3192345"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.491719 12049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:46.506239 12049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:46.510262 12049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:46.510560 12049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:46.511519 12049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:46.512941 12049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:46.513065 12049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.513211 12049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:46.513285 12049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.546368 12049 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:42327
I20260928 15:02:46.546416 12176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:42327 every 8 connection(s)
I20260928 15:02:46.547665 12049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:46.555289 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12049
I20260928 15:02:46.555495 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:46.555943 9267 external_mini_cluster.cc:1428] Setting key 2139f49b6cd462f64538f858bd87effa
Sep 28 15:02:46 dist-test-slave-dmhs krb5kdc[11675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:46.568058 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:46.550159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:56209 (local address 127.9.12.254:37737)
0928 15:02:46.550326 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:46.550331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:46.551443 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 15:02:46.553355 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:46.553361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:46.553482 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:46.553594 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:46.555079 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.555908 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.557851 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.558263 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.558394 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:46.562355 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:46.562360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:46.562363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:46.562395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:46.565147 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:46.565838 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:46.565849 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:46.565851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:46.565980 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:46.566411 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:46.566420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:46.566422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:46.566686 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:46.566845 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:46.567475 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 15:02:46.567701 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:02:46.568988 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:46.550592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37737 (local address 127.9.12.195:56209)
0928 15:02:46.551295 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:46.551331 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:46.553093 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:46.553838 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:46.553847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:46.554346 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:46.554888 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:46.554905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.556064 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.556070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:46.557713 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:46.557725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.558481 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.558491 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:46.558835 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:46.559579 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:46.559613 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:46.562149 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:46.565293 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:46.565302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:46.565316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:46.565710 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:46.566091 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:46.566096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:46.566100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:46.566292 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:46.566815 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:46.566823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:46.567146 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 15:02:46.567387 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 15:02:46.567878 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":288,"threads_started":1}
I20260928 15:02:46.570509 12177 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37737
I20260928 15:02:46.570848 12177 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:46.571617 12177 heartbeater.cc:511] Master 127.9.12.254:37737 requested a full tablet report, sending...
I20260928 15:02:46.572993 11719 ts_manager.cc:194] Registered new tserver with Master: 778041d62c214b8da3b8447b4629085e (127.9.12.195:42327)
I20260928 15:02:46.573832 11719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:56209
I20260928 15:02:46.583853 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:46.600613 11910 heartbeater.cc:503] Master 127.9.12.254:37737 was elected leader, sending a full tablet report...
W20260928 15:02:46.615401 11912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:46.613217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.9.12.254:37737)
0928 15:02:46.613385 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:46.613389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:46.613811 (+ 422us) server_negotiation.cc:407] Connection header received
0928 15:02:46.613953 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:46.613957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:46.614027 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:46.614109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:46.615219 (+ 1110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32894: BlockingRecv error: recv got EOF from 127.0.0.1:32894 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:02:46.619370 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11781
W20260928 15:02:46.622140 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11781: Not authorized: Client connection negotiation failed: client connection to 127.9.12.193:37627: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:02:46.622341 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 11781
I20260928 15:02:46.638279 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11916
W20260928 15:02:46.640913 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11916: Not authorized: Client connection negotiation failed: client connection to 127.9.12.194:43355: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:02:46.641117 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 11916
I20260928 15:02:46.656561 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12049
W20260928 15:02:46.659663 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12049: Not authorized: Client connection negotiation failed: client connection to 127.9.12.195:42327: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:02:46.659821 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12049
I20260928 15:02:46.675190 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 11688
I20260928 15:02:46.676506 11750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:46.779551 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 11688
2026-09-28T15:02:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3480 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:46 dist-test-slave-dmhs krb5kdc[12194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:46 dist-test-slave-dmhs krb5kdc[12194](info): set up 2 sockets
Sep 28 15:02:46 dist-test-slave-dmhs krb5kdc[12194](info): commencing operation
krb5kdc: starting...
W20260928 15:02:47.848933 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:47 dist-test-slave-dmhs krb5kdc[12194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:48.037638 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34581
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:34581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:48.347875 12207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:48.348148 12207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:48.348361 12207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:48.355409 12207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:48.355509 12207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:48.355628 12207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:48.355724 12207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:48.355804 12207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:48.365634 12207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:34581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:48.366966 12207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:48.368279 12207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:48.379568 12212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:48.379519 12215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:48.379614 12213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:48.380447 12207 server_base.cc:1034] running on GCE node
I20260928 15:02:48.381155 12207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:48.382536 12207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:48.383752 12207 hybrid_clock.cc:648] HybridClock initialized: now 1790607768383696 us; error 45 us; skew 500 ppm
I20260928 15:02:48.384104 12207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:48 dist-test-slave-dmhs krb5kdc[12194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607768, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:48.387128 12207 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:48.388621 12207 webserver.cc:460] Webserver started at http://127.9.12.254:38079/ using document root <none> and password file <none>
I20260928 15:02:48.389261 12207 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:48.389430 12207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:48.389731 12207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:48.392714 12207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "70b7dc48143f4a0db6806a3a51a8bd77"
format_stamp: "Formatted at 2026-09-28 15:02:48 on dist-test-slave-dmhs"
server_key: "a8c0be77b1a0092cac4861101f61978b"
server_key_iv: "cc4b1d049f36c061d9bad30c95079399"
server_key_version: "encryptionkey@0"
I20260928 15:02:48.393496 12207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "70b7dc48143f4a0db6806a3a51a8bd77"
format_stamp: "Formatted at 2026-09-28 15:02:48 on dist-test-slave-dmhs"
server_key: "a8c0be77b1a0092cac4861101f61978b"
server_key_iv: "cc4b1d049f36c061d9bad30c95079399"
server_key_version: "encryptionkey@0"
I20260928 15:02:48.398311 12207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:02:48.401479 12222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:48.402731 12207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:48.402909 12207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "70b7dc48143f4a0db6806a3a51a8bd77"
format_stamp: "Formatted at 2026-09-28 15:02:48 on dist-test-slave-dmhs"
server_key: "a8c0be77b1a0092cac4861101f61978b"
server_key_iv: "cc4b1d049f36c061d9bad30c95079399"
server_key_version: "encryptionkey@0"
I20260928 15:02:48.403129 12207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:48.427414 12207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:48.431154 12207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:48.431437 12207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:48.450433 12207 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:34581
I20260928 15:02:48.450502 12273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:34581 every 8 connection(s)
I20260928 15:02:48.452088 12207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:48.453706 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12207
I20260928 15:02:48.453949 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:48.454746 9267 external_mini_cluster.cc:1428] Setting key 82ea945d9b8a230686624b3a354bbda1
I20260928 15:02:48.456909 12274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:48.471764 12274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77: Bootstrap starting.
I20260928 15:02:48.474925 12274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77: Neither blocks nor log segments found. Creating new log.
Sep 28 15:02:48 dist-test-slave-dmhs krb5kdc[12194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:48.475981 12274 log.cc:824] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:48.478803 12274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77: No bootstrap required, opened a new log
I20260928 15:02:48.483144 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:48.457362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.9.12.254:34581)
0928 15:02:48.464734 (+ 7372us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:48.464746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:48.464789 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:02:48.466260 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:48.466284 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:48.466653 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:48.467039 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:48.468587 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:48.469849 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:48.471113 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:48.471763 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:48.471983 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:48.476146 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:48.476161 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:48.476174 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:48.476214 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:48.479243 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:48.479890 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:48.479896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:48.479902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:48.480064 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:48.480520 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:48.480524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:48.480526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:48.481026 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:48.481382 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:48.481767 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 15:02:48.482209 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7170,"thread_start_us":6958,"threads_started":1}
I20260928 15:02:48.486725 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:48.486757 12274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34581 } }
I20260928 15:02:48.487159 12274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:48.487278 12274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b7dc48143f4a0db6806a3a51a8bd77, State: Initialized, Role: FOLLOWER
I20260928 15:02:48.487810 12274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34581 } }
I20260928 15:02:48.488054 12274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:48.488186 12274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:48.488389 12274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:48.490404 12274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34581 } }
I20260928 15:02:48.490883 12274 leader_election.cc:302] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70b7dc48143f4a0db6806a3a51a8bd77; no voters:
I20260928 15:02:48.491518 12274 leader_election.cc:288] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:48.491624 12279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:48.493434 12279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [term 1 LEADER]: Becoming Leader. State: Replica: 70b7dc48143f4a0db6806a3a51a8bd77, State: Running, Role: LEADER
I20260928 15:02:48.493960 12274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:48.493986 12279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34581 } }
I20260928 15:02:48.507210 12281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b7dc48143f4a0db6806a3a51a8bd77. Latest consensus state: current_term: 1 leader_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34581 } } }
I20260928 15:02:48.508148 12281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:48.507666 12280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7dc48143f4a0db6806a3a51a8bd77" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34581 } } }
I20260928 15:02:48.508646 12280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:48.511662 12296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:48.511782 12296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:48.511947 12297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:48.513660 12297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:48.523233 12297 catalog_manager.cc:1347] Generated new cluster ID: b374c5cec8f64265adb1c66c54b3c5e2
I20260928 15:02:48.523360 12297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:48.535462 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:48.526467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.9.12.254:34581)
0928 15:02:48.526633 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:48.526641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:48.527230 (+ 589us) server_negotiation.cc:407] Connection header received
0928 15:02:48.527396 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:48.527406 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:48.527493 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:48.527601 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:48.528776 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:48.529939 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:48.531666 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:48.532376 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:48.532527 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:48.533642 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:48.533649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:48.533654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:48.534471 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:48.534554 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:48.534587 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:48.534698 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:48.534827 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:02:48.534920 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 15:02:48.542963 12297 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:48.544768 12297 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:48.559669 12297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70b7dc48143f4a0db6806a3a51a8bd77: Generated new TSK 0
I20260928 15:02:48.560577 12297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:48.573068 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12207
I20260928 15:02:48.574612 12269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:48.665438 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12207
2026-09-28T15:02:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1877 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:48 dist-test-slave-dmhs krb5kdc[12309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:48 dist-test-slave-dmhs krb5kdc[12309](info): set up 2 sockets
Sep 28 15:02:48 dist-test-slave-dmhs krb5kdc[12309](info): commencing operation
krb5kdc: starting...
W20260928 15:02:49.724903 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:49 dist-test-slave-dmhs krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:49.919569 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37615
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:37615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:50.238809 12322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:50.239147 12322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:50.239306 12322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:50.246618 12322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:50.246795 12322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:50.246912 12322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:50.247049 12322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:50.247117 12322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:50.257061 12322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:37615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:50.258467 12322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:50.259802 12322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:50.271456 12328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.271484 12327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.271557 12330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:50.272389 12322 server_base.cc:1034] running on GCE node
I20260928 15:02:50.273157 12322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:50.274535 12322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:50.275770 12322 hybrid_clock.cc:648] HybridClock initialized: now 1790607770275739 us; error 36 us; skew 500 ppm
I20260928 15:02:50.276127 12322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:50 dist-test-slave-dmhs krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:50.279333 12322 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:50.280773 12322 webserver.cc:460] Webserver started at http://127.9.12.254:40929/ using document root <none> and password file <none>
I20260928 15:02:50.281401 12322 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:50.281504 12322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:50.281749 12322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:50.284677 12322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "f337a5813a594803aad8a291b9a5f85a"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-dmhs"
server_key: "9cb78adf7bd2b1850af6104025752a30"
server_key_iv: "0acd880720ddedaf3f03bdacbde70d56"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.285389 12322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "f337a5813a594803aad8a291b9a5f85a"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-dmhs"
server_key: "9cb78adf7bd2b1850af6104025752a30"
server_key_iv: "0acd880720ddedaf3f03bdacbde70d56"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.290439 12322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:02:50.293723 12337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.295017 12322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:02:50.295217 12322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "f337a5813a594803aad8a291b9a5f85a"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-dmhs"
server_key: "9cb78adf7bd2b1850af6104025752a30"
server_key_iv: "0acd880720ddedaf3f03bdacbde70d56"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.295405 12322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:50.323091 12322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:50.327571 12322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:50.327883 12322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:50.345548 12322 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:37615
I20260928 15:02:50.345582 12388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:37615 every 8 connection(s)
I20260928 15:02:50.346935 12322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:50.349422 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12322
I20260928 15:02:50.349637 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:50.350471 9267 external_mini_cluster.cc:1428] Setting key b69da0f551f89baf20dc3a6a0f5f001a
I20260928 15:02:50.351379 12389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:50.365506 12389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a: Bootstrap starting.
Sep 28 15:02:50 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:50.368745 12389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:50.369800 12389 log.cc:824] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:50.372668 12389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a: No bootstrap required, opened a new log
I20260928 15:02:50.374078 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.352579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.9.12.254:37615)
0928 15:02:50.353362 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:50.353373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:50.353413 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:02:50.355707 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:50.355726 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:50.356109 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:50.356482 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:50.358227 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.359462 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.361267 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.361941 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.362084 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.366843 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:50.366850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:50.366866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:50.366915 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:50.370022 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.370718 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.370725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.370731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.370899 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.371292 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.371296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.371298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.371794 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:50.372114 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:50.372580 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 15:02:50.373138 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":355,"threads_started":1}
I20260928 15:02:50.380116 12389 raft_consensus.cc:348] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f337a5813a594803aad8a291b9a5f85a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37615 } }
I20260928 15:02:50.380590 12389 raft_consensus.cc:374] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:50.380698 12389 raft_consensus.cc:729] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f337a5813a594803aad8a291b9a5f85a, State: Initialized, Role: FOLLOWER
I20260928 15:02:50.381238 12389 consensus_queue.cc:260] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f337a5813a594803aad8a291b9a5f85a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37615 } }
I20260928 15:02:50.381479 12389 raft_consensus.cc:388] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:50.381613 12389 raft_consensus.cc:482] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:50.381788 12389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:50.383908 12389 raft_consensus.cc:504] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f337a5813a594803aad8a291b9a5f85a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37615 } }
I20260928 15:02:50.384438 12389 leader_election.cc:302] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f337a5813a594803aad8a291b9a5f85a; no voters:
I20260928 15:02:50.384977 12389 leader_election.cc:288] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:50.385118 12394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:50.385972 12394 raft_consensus.cc:686] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [term 1 LEADER]: Becoming Leader. State: Replica: f337a5813a594803aad8a291b9a5f85a, State: Running, Role: LEADER
I20260928 15:02:50.386523 12394 consensus_queue.cc:237] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f337a5813a594803aad8a291b9a5f85a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37615 } }
I20260928 15:02:50.387307 12389 sys_catalog.cc:564] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:50.389550 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f337a5813a594803aad8a291b9a5f85a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f337a5813a594803aad8a291b9a5f85a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37615 } } }
I20260928 15:02:50.390003 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:50.389631 12396 sys_catalog.cc:455] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f337a5813a594803aad8a291b9a5f85a. Latest consensus state: current_term: 1 leader_uuid: "f337a5813a594803aad8a291b9a5f85a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f337a5813a594803aad8a291b9a5f85a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37615 } } }
I20260928 15:02:50.390146 12396 sys_catalog.cc:458] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:50.391134 12401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:50.396798 12401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:50.405575 12401 catalog_manager.cc:1347] Generated new cluster ID: 67f01506c7e743d29833c0c38cf88349
I20260928 15:02:50.405700 12401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:50.422933 12401 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:50.424226 12401 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:50.429804 12401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f337a5813a594803aad8a291b9a5f85a: Generated new TSK 0
I20260928 15:02:50.430598 12401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:50.477425 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37615
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:50.813514 12415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:50.813839 12415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:50.813973 12415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:50.820770 12415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:50.820865 12415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:50.820988 12415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:50.830279 12415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:50.831457 12415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:50.832695 12415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:50.845700 12420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.845723 12423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.845659 12421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:50.846398 12415 server_base.cc:1034] running on GCE node
I20260928 15:02:50.846971 12415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:50.847886 12415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:50.849095 12415 hybrid_clock.cc:648] HybridClock initialized: now 1790607770849012 us; error 67 us; skew 500 ppm
I20260928 15:02:50.849406 12415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:50 dist-test-slave-dmhs krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:50.856139 12415 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:50.857635 12415 webserver.cc:460] Webserver started at http://127.9.12.193:45661/ using document root <none> and password file <none>
I20260928 15:02:50.858382 12415 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:50.858541 12415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:50.858845 12415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:50.862023 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "3fb75328cd474012abcdc6596de19108"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-dmhs"
server_key: "7de7340f3a234bc58acef17f78450955"
server_key_iv: "c8b90a689cd597349e57d7952efb45d8"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.862840 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "3fb75328cd474012abcdc6596de19108"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-dmhs"
server_key: "7de7340f3a234bc58acef17f78450955"
server_key_iv: "c8b90a689cd597349e57d7952efb45d8"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.868348 12415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:02:50.871680 12430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.873064 12415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:02:50.873245 12415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "3fb75328cd474012abcdc6596de19108"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-dmhs"
server_key: "7de7340f3a234bc58acef17f78450955"
server_key_iv: "c8b90a689cd597349e57d7952efb45d8"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.873457 12415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:50.899348 12415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:50.903421 12415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:50.903822 12415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:50.904898 12415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:50.906152 12415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:50.906224 12415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.906306 12415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:50.906349 12415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.939081 12415 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:40847
I20260928 15:02:50.939126 12542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:40847 every 8 connection(s)
I20260928 15:02:50.940409 12415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:50.946233 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12415
I20260928 15:02:50.946470 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:50.947146 9267 external_mini_cluster.cc:1428] Setting key 57cd1e25100961efa0e4db55526f237f
Sep 28 15:02:50 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:50.961922 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.942828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:53339 (local address 127.9.12.254:37615)
0928 15:02:50.943276 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:50.943282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:50.944287 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 15:02:50.946021 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:50.946027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:50.946126 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:50.946259 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:50.947895 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.948893 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.951410 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.951820 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.951917 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.956192 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:50.956197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:50.956200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:50.956231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:50.958747 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.959666 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.959669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.959672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.959782 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.960242 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.960245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.960247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.960542 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:50.960649 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:50.961516 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 15:02:50.961678 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260928 15:02:50.963591 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.943440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37615 (local address 127.9.12.193:53339)
0928 15:02:50.944110 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:50.944145 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:50.945762 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:50.946497 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:50.946505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:50.947054 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:50.947691 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:50.947709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.949118 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.949122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:50.951241 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:50.951266 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.952058 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.952067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:50.952424 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.953161 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:50.953196 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:50.955964 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:50.958889 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:50.958900 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:50.958919 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:50.959551 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:50.959883 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:50.959890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:50.959897 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:50.960140 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:50.960669 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:50.960679 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:50.961131 (+ 452us) client_negotiation.cc:769] Sending connection context
0928 15:02:50.961438 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 15:02:50.962021 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":272,"threads_started":1}
I20260928 15:02:50.965257 12543 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37615
I20260928 15:02:50.965580 12543 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:50.966403 12543 heartbeater.cc:511] Master 127.9.12.254:37615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:02:50.968657 12354 ts_manager.cc:194] Registered new tserver with Master: 3fb75328cd474012abcdc6596de19108 (127.9.12.193:40847)
I20260928 15:02:50.970227 12354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:53339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:50.995206 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37615
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:51.312002 12549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:51.312364 12549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:51.312523 12549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:51.319485 12549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:51.319638 12549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:51.319818 12549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:51.329411 12549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:51.330693 12549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:51.332036 12549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:51.344767 12554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:51.344707 12557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:51.344734 12555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:51.345891 12549 server_base.cc:1034] running on GCE node
I20260928 15:02:51.346578 12549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:51.347699 12549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:51.348923 12549 hybrid_clock.cc:648] HybridClock initialized: now 1790607771348865 us; error 48 us; skew 500 ppm
I20260928 15:02:51.349257 12549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:51 dist-test-slave-dmhs krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:51.352919 12549 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:51.354370 12549 webserver.cc:460] Webserver started at http://127.9.12.194:35063/ using document root <none> and password file <none>
I20260928 15:02:51.355032 12549 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:51.355172 12549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:51.355463 12549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:51.358232 12549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "3dae6476055849b5b30579ba73c71848"
format_stamp: "Formatted at 2026-09-28 15:02:51 on dist-test-slave-dmhs"
server_key: "2c09bf47a3fedcf3ae5bfdb01be1fecd"
server_key_iv: "ca797fcce05bf1f0558c4d27e3f90a31"
server_key_version: "encryptionkey@0"
I20260928 15:02:51.358968 12549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "3dae6476055849b5b30579ba73c71848"
format_stamp: "Formatted at 2026-09-28 15:02:51 on dist-test-slave-dmhs"
server_key: "2c09bf47a3fedcf3ae5bfdb01be1fecd"
server_key_iv: "ca797fcce05bf1f0558c4d27e3f90a31"
server_key_version: "encryptionkey@0"
I20260928 15:02:51.363888 12549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:02:51.367292 12564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:51.368781 12549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:51.368945 12549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "3dae6476055849b5b30579ba73c71848"
format_stamp: "Formatted at 2026-09-28 15:02:51 on dist-test-slave-dmhs"
server_key: "2c09bf47a3fedcf3ae5bfdb01be1fecd"
server_key_iv: "ca797fcce05bf1f0558c4d27e3f90a31"
server_key_version: "encryptionkey@0"
I20260928 15:02:51.369135 12549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:51.393316 12549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:51.397908 12549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:51.398206 12549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:51.399276 12549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:51.400696 12549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:51.400768 12549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:51.400851 12549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:51.400895 12549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:51.433171 12549 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:41691
I20260928 15:02:51.433254 12676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:41691 every 8 connection(s)
I20260928 15:02:51.434504 12549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:51.443809 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12549
I20260928 15:02:51.443993 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:51.444509 9267 external_mini_cluster.cc:1428] Setting key 0623956d89d4f6d98471d79a31cbd4e7
Sep 28 15:02:51 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:51.453936 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:51.437033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:48713 (local address 127.9.12.254:37615)
0928 15:02:51.437231 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:51.437237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:51.438402 (+ 1165us) server_negotiation.cc:407] Connection header received
0928 15:02:51.440187 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:51.440192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:51.440306 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:51.440425 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:51.441758 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.442562 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.444061 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.444553 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.444692 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:51.448343 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:51.448348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:51.448351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:51.448383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:51.450916 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:51.451619 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:51.451629 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:51.451631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:51.451750 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:51.452179 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:51.452186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:51.452189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:51.452505 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:51.452648 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:51.453350 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 15:02:51.453590 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:02:51.454895 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:51.437565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37615 (local address 127.9.12.194:48713)
0928 15:02:51.438252 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:51.438294 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:51.439905 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:51.440601 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:51.440608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:51.441084 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:51.441614 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:51.441630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.442687 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.442691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:51.443900 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:51.443916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.444792 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.444800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:51.445083 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:51.445822 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:51.445844 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:51.448104 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:51.451058 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:51.451067 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:51.451082 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:51.451494 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:51.451861 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:51.451866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:51.451870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:51.452053 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:51.452643 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:51.452651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:51.452968 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 15:02:51.453272 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 15:02:51.453819 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260928 15:02:51.456365 12677 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37615
I20260928 15:02:51.456661 12677 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:51.457347 12677 heartbeater.cc:511] Master 127.9.12.254:37615 requested a full tablet report, sending...
I20260928 15:02:51.458784 12354 ts_manager.cc:194] Registered new tserver with Master: 3dae6476055849b5b30579ba73c71848 (127.9.12.194:41691)
I20260928 15:02:51.459553 12354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:48713
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:51.489512 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37615
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:51.806782 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:51.807075 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:51.807169 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:51.814299 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:51.814425 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:51.814544 12682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:51.823918 12682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:51.825217 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:51.826434 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:51.839129 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:51.839197 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:51.839169 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:51.839890 12682 server_base.cc:1034] running on GCE node
I20260928 15:02:51.840528 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:51.841441 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:51.842630 12682 hybrid_clock.cc:648] HybridClock initialized: now 1790607771842591 us; error 38 us; skew 500 ppm
I20260928 15:02:51.842932 12682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:51 dist-test-slave-dmhs krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:51.845969 12682 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:51.847462 12682 webserver.cc:460] Webserver started at http://127.9.12.195:35789/ using document root <none> and password file <none>
I20260928 15:02:51.848145 12682 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:51.848292 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:51.848609 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:51.851496 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "7d66ed66030c494dbde32ebde3c4e12c"
format_stamp: "Formatted at 2026-09-28 15:02:51 on dist-test-slave-dmhs"
server_key: "984f3a3a30d032afca1e0ebeb956c63a"
server_key_iv: "c217630a8e1b44eb6595bfe86c660a54"
server_key_version: "encryptionkey@0"
I20260928 15:02:51.852375 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "7d66ed66030c494dbde32ebde3c4e12c"
format_stamp: "Formatted at 2026-09-28 15:02:51 on dist-test-slave-dmhs"
server_key: "984f3a3a30d032afca1e0ebeb956c63a"
server_key_iv: "c217630a8e1b44eb6595bfe86c660a54"
server_key_version: "encryptionkey@0"
I20260928 15:02:51.857810 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:02:51.861464 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:51.862795 12682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:51.862978 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "7d66ed66030c494dbde32ebde3c4e12c"
format_stamp: "Formatted at 2026-09-28 15:02:51 on dist-test-slave-dmhs"
server_key: "984f3a3a30d032afca1e0ebeb956c63a"
server_key_iv: "c217630a8e1b44eb6595bfe86c660a54"
server_key_version: "encryptionkey@0"
I20260928 15:02:51.863197 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:51.892712 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:51.896770 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:51.897080 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:51.898151 12682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:51.899539 12682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:51.899653 12682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:51.899770 12682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:51.899837 12682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:51.932549 12682 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:40215
I20260928 15:02:51.932596 12810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:40215 every 8 connection(s)
I20260928 15:02:51.933851 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:51.938473 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12682
I20260928 15:02:51.938655 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:51.939230 9267 external_mini_cluster.cc:1428] Setting key b26510101afa1885e0342494937cec10
Sep 28 15:02:51 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:51.954691 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:51.936482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:60931 (local address 127.9.12.254:37615)
0928 15:02:51.936690 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:51.936697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:51.937872 (+ 1175us) server_negotiation.cc:407] Connection header received
0928 15:02:51.940794 (+ 2922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:51.940799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:51.940884 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:51.941002 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:51.942887 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.943633 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.945207 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.945603 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.945734 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:51.949105 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:51.949110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:51.949113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:51.949144 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:51.951542 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:51.952293 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:51.952302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:51.952306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:51.952440 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:51.952874 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:51.952879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:51.952883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:51.953165 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:51.953322 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:51.954091 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 15:02:51.954315 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:02:51.955865 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:51.936991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37615 (local address 127.9.12.195:60931)
0928 15:02:51.937712 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:51.937750 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:51.940433 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:51.941262 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:51.941273 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:51.942015 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:51.942716 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:51.942743 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.943777 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.943782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:51.945060 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:51.945074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:51.945836 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:51.945843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:51.946071 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:51.946736 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:51.946757 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:51.948891 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:51.951687 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:51.951695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:51.951719 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:51.952116 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:51.952553 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:51.952558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:51.952570 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:51.952753 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:51.953293 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:51.953300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:51.953722 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 15:02:51.953973 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 15:02:51.954487 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":275,"threads_started":1}
I20260928 15:02:51.957288 12811 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37615
I20260928 15:02:51.957551 12811 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:51.958232 12811 heartbeater.cc:511] Master 127.9.12.254:37615 requested a full tablet report, sending...
I20260928 15:02:51.959600 12354 ts_manager.cc:194] Registered new tserver with Master: 7d66ed66030c494dbde32ebde3c4e12c (127.9.12.195:40215)
I20260928 15:02:51.960465 12354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:60931
I20260928 15:02:51.967514 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:51.973234 12543 heartbeater.cc:503] Master 127.9.12.254:37615 was elected leader, sending a full tablet report...
Sep 28 15:02:51 dist-test-slave-dmhs krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:52.016122 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.002492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.9.12.254:37615)
0928 15:02:52.002667 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.002671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.003208 (+ 537us) server_negotiation.cc:407] Connection header received
0928 15:02:52.003367 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.003371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.003437 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.003530 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.004925 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.005649 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.006816 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.007248 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.007346 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.011072 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.011077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.011081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.011114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.013835 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.014472 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.014477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.014479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.014623 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.014957 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.014961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.014963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.015325 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.015528 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.015731 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.015894 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:02:52.016754 12354 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39466
I20260928 15:02:52.018471 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12415
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:52.033521 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.019626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.9.12.193:40847)
0928 15:02:52.020346 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.020352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.020376 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:52.020428 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.020434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.020646 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.020781 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.021823 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.022802 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.023897 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.024487 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.024580 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.028371 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.028383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.028397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.028434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.031072 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.031644 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.031650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.031653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.031767 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.032065 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.032068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.032070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.032473 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.032682 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.032932 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.033130 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":363,"threads_started":1}
W20260928 15:02:52.034715 12538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55232
W20260928 15:02:52.035748 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12415: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:52.035969 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12415
I20260928 15:02:52.051843 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12549
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:52.067330 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.052782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.9.12.194:41691)
0928 15:02:52.053317 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.053325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.053350 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:52.053410 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.053416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.053620 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.053743 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.054890 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.055914 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.057172 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.057690 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.057782 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.061831 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.061843 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.061857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.061899 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.064750 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.065333 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.065340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.065342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.065463 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.065791 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.065794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.065796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.066147 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.066396 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.066680 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.066918 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":272,"threads_started":1}
W20260928 15:02:52.068833 12672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55364
W20260928 15:02:52.070143 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:52.070267 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12549
I20260928 15:02:52.086166 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12682
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:52.102785 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.087070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.9.12.195:40215)
0928 15:02:52.087700 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.087707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.087735 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:02:52.087793 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.087800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.088109 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.088234 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.089594 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.090613 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.092473 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.093168 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.093264 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.097544 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.097554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.097567 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.097598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.100429 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.101101 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.101107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.101110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.101224 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.101524 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.101527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.101529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.101838 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.102082 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.102310 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.102507 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":320,"threads_started":1}
W20260928 15:02:52.104096 12806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35180
W20260928 15:02:52.105374 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12682: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:02:52.105556 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12682
I20260928 15:02:52.120960 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12322
I20260928 15:02:52.122134 12384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:52.224059 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12322
2026-09-28T15:02:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3566 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12828](info): set up 2 sockets
Sep 28 15:02:52 dist-test-slave-dmhs krb5kdc[12828](info): commencing operation
krb5kdc: starting...
W20260928 15:02:53.292939 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:53 dist-test-slave-dmhs krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:53.484074 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34615
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:53.804160 12841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:53.804482 12841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:53.804666 12841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:53.811872 12841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:53.811988 12841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:53.812093 12841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:53.812172 12841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:53.812254 12841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:53.822088 12841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:53.823439 12841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:53.824815 12841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:53.836541 12846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:53.836624 12849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:53.836586 12847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:53.837627 12841 server_base.cc:1034] running on GCE node
I20260928 15:02:53.838442 12841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:53.839829 12841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:53.841022 12841 hybrid_clock.cc:648] HybridClock initialized: now 1790607773840971 us; error 39 us; skew 500 ppm
I20260928 15:02:53.841322 12841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:53 dist-test-slave-dmhs krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:53.844391 12841 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:53.845875 12841 webserver.cc:460] Webserver started at http://127.9.12.254:44497/ using document root <none> and password file <none>
I20260928 15:02:53.846567 12841 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:53.846698 12841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:53.847045 12841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:53.850039 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "ff7be070ee594c5ba23f92457d735227"
format_stamp: "Formatted at 2026-09-28 15:02:53 on dist-test-slave-dmhs"
server_key: "05e25ebaff2b5d497933e3222ad6bea2"
server_key_iv: "242018a75392e0622c4d0408909c816f"
server_key_version: "encryptionkey@0"
I20260928 15:02:53.850807 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "ff7be070ee594c5ba23f92457d735227"
format_stamp: "Formatted at 2026-09-28 15:02:53 on dist-test-slave-dmhs"
server_key: "05e25ebaff2b5d497933e3222ad6bea2"
server_key_iv: "242018a75392e0622c4d0408909c816f"
server_key_version: "encryptionkey@0"
I20260928 15:02:53.855979 12841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:02:53.859638 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:53.861033 12841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:53.861219 12841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "ff7be070ee594c5ba23f92457d735227"
format_stamp: "Formatted at 2026-09-28 15:02:53 on dist-test-slave-dmhs"
server_key: "05e25ebaff2b5d497933e3222ad6bea2"
server_key_iv: "242018a75392e0622c4d0408909c816f"
server_key_version: "encryptionkey@0"
I20260928 15:02:53.861387 12841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:53.882611 12841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:53.886684 12841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:53.887001 12841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:53.905524 12841 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:34615
I20260928 15:02:53.905563 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:34615 every 8 connection(s)
I20260928 15:02:53.906917 12841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:53.910753 12908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:53.912753 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12841
I20260928 15:02:53.913008 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:53.913805 9267 external_mini_cluster.cc:1428] Setting key 2fc87490d50177635319c90800fc9488
I20260928 15:02:53.923698 12908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227: Bootstrap starting.
I20260928 15:02:53.927711 12908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227: Neither blocks nor log segments found. Creating new log.
Sep 28 15:02:53 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:53.928948 12908 log.cc:824] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:53.932663 12908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227: No bootstrap required, opened a new log
I20260928 15:02:53.938822 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:53.915910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.9.12.254:34615)
0928 15:02:53.916780 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:53.916791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:53.916838 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:02:53.918554 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:53.918579 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:53.918967 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:53.919290 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:53.921195 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:53.922566 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:53.924386 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:53.925048 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:53.925199 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:53.930145 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:53.930154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:53.930169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:53.930214 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:53.934154 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:53.934892 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:53.934902 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:53.934910 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:53.935119 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:53.935481 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:53.935490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:53.935494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:53.936222 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:53.936577 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:53.936907 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:02:53.937380 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":430,"threads_started":1}
I20260928 15:02:53.943028 12908 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7be070ee594c5ba23f92457d735227" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34615 } }
I20260928 15:02:53.943404 12908 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:53.943528 12908 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7be070ee594c5ba23f92457d735227, State: Initialized, Role: FOLLOWER
I20260928 15:02:53.944101 12908 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7be070ee594c5ba23f92457d735227" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34615 } }
I20260928 15:02:53.944372 12908 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:53.944527 12908 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:53.944700 12908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:53.947053 12908 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7be070ee594c5ba23f92457d735227" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34615 } }
I20260928 15:02:53.947500 12908 leader_election.cc:302] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7be070ee594c5ba23f92457d735227; no voters:
I20260928 15:02:53.948171 12908 leader_election.cc:288] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:53.948350 12913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:53.949038 12913 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [term 1 LEADER]: Becoming Leader. State: Replica: ff7be070ee594c5ba23f92457d735227, State: Running, Role: LEADER
I20260928 15:02:53.949611 12913 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7be070ee594c5ba23f92457d735227" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34615 } }
I20260928 15:02:53.950526 12908 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:53.952523 12914 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7be070ee594c5ba23f92457d735227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7be070ee594c5ba23f92457d735227" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34615 } } }
I20260928 15:02:53.952685 12915 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7be070ee594c5ba23f92457d735227. Latest consensus state: current_term: 1 leader_uuid: "ff7be070ee594c5ba23f92457d735227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7be070ee594c5ba23f92457d735227" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34615 } } }
I20260928 15:02:53.952942 12914 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:53.952952 12915 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:53.953874 12920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:53.960439 12920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:53.971621 12920 catalog_manager.cc:1347] Generated new cluster ID: ba597969a9434da4aa604b3609ad8dec
I20260928 15:02:53.971767 12920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:53.986577 12920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:53.988014 12920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:53.998538 12920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7be070ee594c5ba23f92457d735227: Generated new TSK 0
I20260928 15:02:53.999622 12920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:54.063282 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34615
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:54.392074 12934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:54.392432 12934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:54.392613 12934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:54.399819 12934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:54.399922 12934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:54.400105 12934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:54.409811 12934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:54.411142 12934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:54.412519 12934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:54.425885 12939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.425783 12942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.425864 12940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:54.426616 12934 server_base.cc:1034] running on GCE node
I20260928 15:02:54.427167 12934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:54.428068 12934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:54.429261 12934 hybrid_clock.cc:648] HybridClock initialized: now 1790607774429201 us; error 53 us; skew 500 ppm
I20260928 15:02:54.429574 12934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:54 dist-test-slave-dmhs krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:54.432786 12934 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:54.434218 12934 webserver.cc:460] Webserver started at http://127.9.12.193:36907/ using document root <none> and password file <none>
I20260928 15:02:54.434881 12934 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:54.435029 12934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:54.435338 12934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:54.438231 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "4f9002fbf54e40f985b8cf20cce96e89"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-dmhs"
server_key: "8e3921a6ba8c462587c62ca14e04027f"
server_key_iv: "7a24cf5fbc4424485ba46f9a3b65f687"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.439009 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "4f9002fbf54e40f985b8cf20cce96e89"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-dmhs"
server_key: "8e3921a6ba8c462587c62ca14e04027f"
server_key_iv: "7a24cf5fbc4424485ba46f9a3b65f687"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.443852 12934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 15:02:54.448159 12949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.449936 12934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:02:54.450112 12934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "4f9002fbf54e40f985b8cf20cce96e89"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-dmhs"
server_key: "8e3921a6ba8c462587c62ca14e04027f"
server_key_iv: "7a24cf5fbc4424485ba46f9a3b65f687"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.450381 12934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:54.469696 12934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:54.474025 12934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:54.474359 12934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:54.475535 12934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:54.476945 12934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:54.477018 12934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.477198 12934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:54.477259 12934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.510210 12934 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:40435
I20260928 15:02:54.510229 13061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:40435 every 8 connection(s)
I20260928 15:02:54.511507 12934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:54.515251 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 12934
I20260928 15:02:54.515492 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:54.516187 9267 external_mini_cluster.cc:1428] Setting key a4130b8c90a66c0fadec068b642e2855
Sep 28 15:02:54 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:54.533545 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:54.514097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:43235 (local address 127.9.12.254:34615)
0928 15:02:54.514551 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:54.514556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:54.515557 (+ 1001us) server_negotiation.cc:407] Connection header received
0928 15:02:54.517611 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:54.517617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:54.517710 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:54.517840 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:54.520259 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.521024 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.523510 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.523902 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.523996 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:54.527809 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:54.527814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:54.527818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:54.527850 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:54.530396 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:54.531121 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:54.531131 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:54.531133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:54.531249 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:54.531753 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:54.531759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:54.531763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:54.532041 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:54.532197 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:54.532934 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 15:02:54.533181 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":255,"threads_started":1}
I20260928 15:02:54.534502 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:54.514532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34615 (local address 127.9.12.193:43235)
0928 15:02:54.515378 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:54.515434 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:54.517351 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:54.518388 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:54.518400 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:54.519207 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:54.520076 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:54.520105 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.521196 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.521204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:54.523360 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:54.523378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.524138 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.524147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:54.524465 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:54.525155 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:54.525177 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:54.527574 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:54.530546 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:54.530555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:54.530573 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:54.530990 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:54.531370 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:54.531374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:54.531378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:54.531649 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:54.532170 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:54.532178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:54.532572 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 15:02:54.532846 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:02:54.533376 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:02:54.536098 13062 heartbeater.cc:347] Connected to a master server at 127.9.12.254:34615
I20260928 15:02:54.536474 13062 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:54.537242 13062 heartbeater.cc:511] Master 127.9.12.254:34615 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:02:54.539675 12873 ts_manager.cc:194] Registered new tserver with Master: 4f9002fbf54e40f985b8cf20cce96e89 (127.9.12.193:40435)
I20260928 15:02:54.541828 12873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:43235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:54.562862 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34615
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:54.875722 13068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:54.876017 13068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:54.876228 13068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:54.883519 13068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:54.883625 13068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:54.883807 13068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:54.893361 13068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:54.894649 13068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:54.896026 13068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:54.908964 13073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.909144 13074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.909663 13076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:54.910372 13068 server_base.cc:1034] running on GCE node
I20260928 15:02:54.910974 13068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:54.911917 13068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:54.913154 13068 hybrid_clock.cc:648] HybridClock initialized: now 1790607774913098 us; error 48 us; skew 500 ppm
I20260928 15:02:54.913484 13068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:54 dist-test-slave-dmhs krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:54.916719 13068 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:54.918255 13068 webserver.cc:460] Webserver started at http://127.9.12.194:42531/ using document root <none> and password file <none>
I20260928 15:02:54.918975 13068 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:54.919126 13068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:54.919437 13068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:54.922437 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "71ad010a42f44549a61e33d3438da6a8"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-dmhs"
server_key: "9941375d4e628df30276980e06b48ee8"
server_key_iv: "f666520dbbc5376d447a880444337263"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.923243 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "71ad010a42f44549a61e33d3438da6a8"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-dmhs"
server_key: "9941375d4e628df30276980e06b48ee8"
server_key_iv: "f666520dbbc5376d447a880444337263"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.928531 13068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:02:54.931982 13083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.933347 13068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:54.933535 13068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "71ad010a42f44549a61e33d3438da6a8"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-dmhs"
server_key: "9941375d4e628df30276980e06b48ee8"
server_key_iv: "f666520dbbc5376d447a880444337263"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.933754 13068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:54.966535 13068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:54.971305 13068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:54.971618 13068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:54.972669 13068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:54.974066 13068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:54.974212 13068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.974387 13068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:54.974483 13068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.008098 13068 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:42161
I20260928 15:02:55.008147 13195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:42161 every 8 connection(s)
I20260928 15:02:55.009512 13068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:55.011415 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13068
I20260928 15:02:55.011626 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:55.012084 9267 external_mini_cluster.cc:1428] Setting key b36b1d776448a7d9285cb2242c9ea4c2
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:02:55.031756 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.012431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:35213 (local address 127.9.12.254:34615)
0928 15:02:55.012645 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.012651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.014093 (+ 1442us) server_negotiation.cc:407] Connection header received
0928 15:02:55.015956 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.015962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.016078 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.016184 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.017669 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.018392 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.020820 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.021298 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.021400 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.025696 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.025701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.025705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.025740 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.028587 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.029388 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.029398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.029400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.029534 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.029982 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.029993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.029996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.030261 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.030419 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.031150 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.031392 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:02:55.033006 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.013039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34615 (local address 127.9.12.194:35213)
0928 15:02:55.013919 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.013974 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.015687 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.016442 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.016450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.016948 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:55.017504 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.017521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.018563 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.018567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.020662 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.020680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.021538 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.021547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.021877 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.022623 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:55.022662 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:55.025437 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:55.028750 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.028759 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.028777 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.029246 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.029652 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.029657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.029661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.029859 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.030393 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:55.030401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:55.030771 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.031054 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.031622 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
I20260928 15:02:55.034606 13196 heartbeater.cc:347] Connected to a master server at 127.9.12.254:34615
I20260928 15:02:55.034952 13196 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:55.035881 13196 heartbeater.cc:511] Master 127.9.12.254:34615 requested a full tablet report, sending...
I20260928 15:02:55.037678 12873 ts_manager.cc:194] Registered new tserver with Master: 71ad010a42f44549a61e33d3438da6a8 (127.9.12.194:42161)
I20260928 15:02:55.038725 12873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:35213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:55.058931 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34615
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:55.379146 13201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:55.379410 13201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:55.379499 13201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:55.386456 13201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:55.386552 13201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:55.386713 13201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:55.396296 13201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:55.397514 13201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:55.398761 13201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:55.411722 13207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.411664 13206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.411742 13209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:55.412475 13201 server_base.cc:1034] running on GCE node
I20260928 15:02:55.413054 13201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:55.413970 13201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:55.415200 13201 hybrid_clock.cc:648] HybridClock initialized: now 1790607775415163 us; error 52 us; skew 500 ppm
I20260928 15:02:55.415532 13201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:55.418658 13201 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:55.419981 13201 webserver.cc:460] Webserver started at http://127.9.12.195:44265/ using document root <none> and password file <none>
I20260928 15:02:55.420722 13201 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:55.420867 13201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:55.421195 13201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:55.423942 13201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "1b193e59e2534b5c9cfaedd731dc4421"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-dmhs"
server_key: "f78fb03c3194b2aa52061ec7892ab6b7"
server_key_iv: "b8d4e2a9b12e9a73912ea8eb5a83b6c0"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.424722 13201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "1b193e59e2534b5c9cfaedd731dc4421"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-dmhs"
server_key: "f78fb03c3194b2aa52061ec7892ab6b7"
server_key_iv: "b8d4e2a9b12e9a73912ea8eb5a83b6c0"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.429587 13201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 15:02:55.432817 13216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.433991 13201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:02:55.434165 13201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "1b193e59e2534b5c9cfaedd731dc4421"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-dmhs"
server_key: "f78fb03c3194b2aa52061ec7892ab6b7"
server_key_iv: "b8d4e2a9b12e9a73912ea8eb5a83b6c0"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.434368 13201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:55.447657 13201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:55.451745 13201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:55.452024 13201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:55.452952 13201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:55.454190 13201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:55.454299 13201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.454440 13201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:55.454514 13201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.486964 13201 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:37757
I20260928 15:02:55.487011 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:37757 every 8 connection(s)
I20260928 15:02:55.488200 13201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:55.498479 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13201
I20260928 15:02:55.498718 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:55.499181 9267 external_mini_cluster.cc:1428] Setting key dda59a161bbe9880782c34eda3009c9d
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:55.506899 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.490711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:49765 (local address 127.9.12.254:34615)
0928 15:02:55.490901 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.490905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.491969 (+ 1064us) server_negotiation.cc:407] Connection header received
0928 15:02:55.493706 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.493711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.493779 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.493865 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.495227 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.495956 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.497485 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.497889 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.497983 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.501587 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.501593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.501597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.501635 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.504457 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.505070 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.505073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.505075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.505200 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.505555 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.505558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.505560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.505802 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.505905 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.506489 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.506648 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:02:55.507822 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.491131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34615 (local address 127.9.12.195:49765)
0928 15:02:55.491808 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.491849 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.493456 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.494059 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.494067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.494552 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:55.495087 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.495102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.496086 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.496089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.497368 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.497379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.498061 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.498065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.498281 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.498999 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:55.499021 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:55.501399 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:55.504567 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.504573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.504584 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.504951 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.505291 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.505294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.505296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.505457 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.505898 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:55.505903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:55.506198 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.506392 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.506725 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":267,"threads_started":1}
I20260928 15:02:55.509130 13329 heartbeater.cc:347] Connected to a master server at 127.9.12.254:34615
I20260928 15:02:55.509434 13329 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:55.510090 13329 heartbeater.cc:511] Master 127.9.12.254:34615 requested a full tablet report, sending...
I20260928 15:02:55.511384 12873 ts_manager.cc:194] Registered new tserver with Master: 1b193e59e2534b5c9cfaedd731dc4421 (127.9.12.195:37757)
I20260928 15:02:55.512133 12873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:49765
I20260928 15:02:55.514475 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 15:02:55.545308 13062 heartbeater.cc:503] Master 127.9.12.254:34615 was elected leader, sending a full tablet report...
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:55.562001 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.547406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.9.12.193:40435)
0928 15:02:55.547992 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.547998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.548022 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:55.548085 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.548090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.548350 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.548467 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.549710 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.550785 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.552070 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.552711 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.552808 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.556623 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.556633 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.556647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.556676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.559494 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.560088 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.560094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.560096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.560206 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.560600 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.560604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.560605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.560910 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.561126 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.561323 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.561610 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":305,"threads_started":1}
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:55.579103 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.566102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.9.12.254:34615)
0928 15:02:55.566301 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.566308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.566569 (+ 261us) server_negotiation.cc:407] Connection header received
0928 15:02:55.566752 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.566764 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.566827 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.566942 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.568108 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.568899 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.570127 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.570546 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.570683 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.574505 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.574509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.574512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.574543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.577022 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.577590 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.577593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.577595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.577713 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.578013 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.578016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.578018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.578302 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.578479 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.578695 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.578847 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 15:02:55.579638 12873 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57596
I20260928 15:02:55.580726 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12934
I20260928 15:02:55.592628 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.581598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.9.12.193:40435)
0928 15:02:55.581773 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.581777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.582145 (+ 368us) server_negotiation.cc:407] Connection header received
0928 15:02:55.582260 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.582264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.582328 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.582405 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.583496 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.584150 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.585342 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.585749 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.585851 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.587947 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.587951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.587955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.587984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.590593 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.591116 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.591119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.591121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.591236 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.591525 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.591528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.591530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.591761 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.591934 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.592173 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.592392 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:02:55.593191 13057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:55.674113 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12934
I20260928 15:02:55.689878 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13068
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:55.705945 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.690812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.9.12.194:42161)
0928 15:02:55.691308 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.691315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.691338 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:02:55.691401 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.691407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.691625 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.691798 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.693089 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.694147 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.695425 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.695950 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.696090 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.700343 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.700352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.700365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.700402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.702957 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.703604 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.703610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.703613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.703748 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.704130 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.704135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.704140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.704559 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.704883 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.705240 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.705513 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":266,"threads_started":1}
I20260928 15:02:55.707188 13191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:55.788245 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13068
I20260928 15:02:55.804988 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13201
Sep 28 15:02:55 dist-test-slave-dmhs krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:55.820600 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.806006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.9.12.195:37757)
0928 15:02:55.806657 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.806664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.806692 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:02:55.806751 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.806758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.806973 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.807099 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.808325 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.809355 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.810586 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.811134 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.811233 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.815689 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.815712 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.815725 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.815755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.818211 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.818778 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.818784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.818786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.818893 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.819182 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.819185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.819187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.819490 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.819730 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.819977 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.820190 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260928 15:02:55.821815 13324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:55.905817 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13201
I20260928 15:02:55.926535 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 12841
I20260928 15:02:55.927666 12903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:56.027166 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 12841
2026-09-28T15:02:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3802 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:56 dist-test-slave-dmhs krb5kdc[13348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:56 dist-test-slave-dmhs krb5kdc[13348](info): set up 2 sockets
Sep 28 15:02:56 dist-test-slave-dmhs krb5kdc[13348](info): commencing operation
krb5kdc: starting...
W20260928 15:02:57.108906 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:02:57 dist-test-slave-dmhs krb5kdc[13348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:02:57.297995 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40161
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:57.612145 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:57.612473 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:57.612679 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:57.619784 13361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:57.619875 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:57.619966 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:57.620056 13361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:57.620148 13361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:57.629742 13361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:57.630973 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:57.632306 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:57.643847 13369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.643952 13367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.643990 13366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:57.645169 13361 server_base.cc:1034] running on GCE node
I20260928 15:02:57.645925 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:57.647243 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:57.648464 13361 hybrid_clock.cc:648] HybridClock initialized: now 1790607777648432 us; error 47 us; skew 500 ppm
I20260928 15:02:57.648797 13361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:57 dist-test-slave-dmhs krb5kdc[13348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:57.651696 13361 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:02:57.652887 13361 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:57.652976 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:57.653235 13361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:57.656144 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "5793ea6ad5194196a9fe26e2ada10c58"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-dmhs"
server_key: "f42d79b91943d99e96d91c1c9545cced"
server_key_iv: "a4e22e7e9b3dbe54e28fdea56eb1b790"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.656945 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "5793ea6ad5194196a9fe26e2ada10c58"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-dmhs"
server_key: "f42d79b91943d99e96d91c1c9545cced"
server_key_iv: "a4e22e7e9b3dbe54e28fdea56eb1b790"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.661588 13361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:02:57.664692 13375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:57.665923 13361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:02:57.666102 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "5793ea6ad5194196a9fe26e2ada10c58"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-dmhs"
server_key: "f42d79b91943d99e96d91c1c9545cced"
server_key_iv: "a4e22e7e9b3dbe54e28fdea56eb1b790"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.666298 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:57.686138 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:57.689869 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:57.690112 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:57.707683 13361 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:40161
I20260928 15:02:57.707701 13427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:40161 every 8 connection(s)
I20260928 15:02:57.708561 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:57.711958 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:57.716559 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13361
I20260928 15:02:57.716777 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:02:57.717375 9267 external_mini_cluster.cc:1428] Setting key de0753933369f3b4bcf33636bf6fe6c7
I20260928 15:02:57.722230 13428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58: Bootstrap starting.
I20260928 15:02:57.725900 13428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:57.727173 13428 log.cc:824] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:57.730254 13428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58: No bootstrap required, opened a new log
Sep 28 15:02:57 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:57.737982 13428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793ea6ad5194196a9fe26e2ada10c58" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40161 } }
I20260928 15:02:57.738368 13428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:57.738497 13428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5793ea6ad5194196a9fe26e2ada10c58, State: Initialized, Role: FOLLOWER
I20260928 15:02:57.739207 13428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793ea6ad5194196a9fe26e2ada10c58" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40161 } }
I20260928 15:02:57.739466 13428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:57.739593 13428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:57.739769 13428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:57.740769 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.719220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.9.12.254:40161)
0928 15:02:57.720119 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.720129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.720169 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:02:57.721884 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.721905 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.722272 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.722622 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:57.724629 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.725929 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.727776 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.728738 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.728859 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.733349 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:57.733356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:57.733373 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:57.733410 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:57.736497 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:57.737149 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:57.737155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:57.737161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:57.737314 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:57.737680 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:57.737685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:57.737689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:57.738213 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:57.738523 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:57.738956 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 15:02:57.739369 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":449,"threads_started":1}
I20260928 15:02:57.742519 13428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793ea6ad5194196a9fe26e2ada10c58" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40161 } }
I20260928 15:02:57.743090 13428 leader_election.cc:302] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5793ea6ad5194196a9fe26e2ada10c58; no voters:
I20260928 15:02:57.743818 13428 leader_election.cc:288] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:57.743932 13433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:57.744591 13433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [term 1 LEADER]: Becoming Leader. State: Replica: 5793ea6ad5194196a9fe26e2ada10c58, State: Running, Role: LEADER
I20260928 15:02:57.745180 13433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793ea6ad5194196a9fe26e2ada10c58" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40161 } }
I20260928 15:02:57.746109 13428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:57.748203 13434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5793ea6ad5194196a9fe26e2ada10c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793ea6ad5194196a9fe26e2ada10c58" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40161 } } }
I20260928 15:02:57.749248 13434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.749524 13435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5793ea6ad5194196a9fe26e2ada10c58. Latest consensus state: current_term: 1 leader_uuid: "5793ea6ad5194196a9fe26e2ada10c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793ea6ad5194196a9fe26e2ada10c58" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40161 } } }
I20260928 15:02:57.749931 13435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.753729 13441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:57.757915 13441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:57.766310 13441 catalog_manager.cc:1347] Generated new cluster ID: 10eb573ad3324d8f8196cfa02847c825
I20260928 15:02:57.766456 13441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:57.785485 13441 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:57.787379 13441 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:57.795841 13441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58: Generated new TSK 0
I20260928 15:02:57.796867 13441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:02:57.840996 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40161
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:58.164819 13454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:58.165073 13454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:58.165246 13454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:58.172744 13454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:58.172870 13454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:58.173060 13454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:02:58.182430 13454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:58.183713 13454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:58.185017 13454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:58.197566 13459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:58.197672 13460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:58.197639 13462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:58.198442 13454 server_base.cc:1034] running on GCE node
I20260928 15:02:58.199011 13454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:58.199929 13454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:58.201148 13454 hybrid_clock.cc:648] HybridClock initialized: now 1790607778201102 us; error 54 us; skew 500 ppm
I20260928 15:02:58.201475 13454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:58 dist-test-slave-dmhs krb5kdc[13348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:58.204514 13454 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:02:58.205593 13454 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:58.205729 13454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:58.206043 13454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:58.208994 13454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "c70e3ecdd2d74202951afdae68cc504d"
format_stamp: "Formatted at 2026-09-28 15:02:58 on dist-test-slave-dmhs"
server_key: "62044e077b2135c76b6f7c9fbb002b17"
server_key_iv: "d685b6ca6b1cd1c348b22ea9c3a4d5c2"
server_key_version: "encryptionkey@0"
I20260928 15:02:58.209775 13454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "c70e3ecdd2d74202951afdae68cc504d"
format_stamp: "Formatted at 2026-09-28 15:02:58 on dist-test-slave-dmhs"
server_key: "62044e077b2135c76b6f7c9fbb002b17"
server_key_iv: "d685b6ca6b1cd1c348b22ea9c3a4d5c2"
server_key_version: "encryptionkey@0"
I20260928 15:02:58.214617 13454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:02:58.217835 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:58.219216 13454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:02:58.219383 13454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "c70e3ecdd2d74202951afdae68cc504d"
format_stamp: "Formatted at 2026-09-28 15:02:58 on dist-test-slave-dmhs"
server_key: "62044e077b2135c76b6f7c9fbb002b17"
server_key_iv: "d685b6ca6b1cd1c348b22ea9c3a4d5c2"
server_key_version: "encryptionkey@0"
I20260928 15:02:58.219631 13454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:58.244427 13454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:58.248389 13454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:58.248668 13454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:58.249554 13454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:58.250902 13454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:58.250995 13454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:58.251109 13454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:58.251169 13454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:58.284482 13454 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:45367
I20260928 15:02:58.284565 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:45367 every 8 connection(s)
I20260928 15:02:58.285329 13454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:58.289703 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13454
I20260928 15:02:58.289928 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:58.290688 9267 external_mini_cluster.cc:1428] Setting key 482e642d510b1fed414556b5912a013d
Sep 28 15:02:58 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:58.307736 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:58.287621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42461 (local address 127.9.12.254:40161)
0928 15:02:58.288080 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:58.288084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:58.288920 (+ 836us) server_negotiation.cc:407] Connection header received
0928 15:02:58.291488 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:58.291494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:58.291587 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:58.291702 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:58.293518 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.294273 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.296738 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.297107 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.297199 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:58.300999 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:58.301007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:58.301015 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:58.301058 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:58.304512 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:58.305231 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:58.305239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:58.305245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:58.305413 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:58.305807 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:58.305815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:58.305821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:58.306228 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:58.306382 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:58.307021 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 15:02:58.307272 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":266,"threads_started":1}
I20260928 15:02:58.308534 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:58.288044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40161 (local address 127.9.12.193:42461)
0928 15:02:58.288775 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:58.288810 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:58.291202 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:58.291939 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:58.291947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:58.292538 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:58.293335 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:58.293365 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.294429 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.294437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:58.296604 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:58.296620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.297314 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.297323 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:58.297625 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:58.298297 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:58.298328 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:58.300761 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:58.304668 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:58.304674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:58.304686 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:58.305099 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:58.305517 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:58.305520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:58.305522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:58.305691 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:58.306345 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:58.306351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:58.306644 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 15:02:58.306854 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 15:02:58.307254 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":335,"threads_started":1}
I20260928 15:02:58.310165 13582 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40161
I20260928 15:02:58.310469 13582 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:58.311129 13582 heartbeater.cc:511] Master 127.9.12.254:40161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:02:58.313328 13392 ts_manager.cc:194] Registered new tserver with Master: c70e3ecdd2d74202951afdae68cc504d (127.9.12.193:45367)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:02:58.315191 13392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:42461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:02:58.337555 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40161
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:58.648923 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:58.649223 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:58.649398 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:58.656551 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:58.656701 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:58.656886 13588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:02:58.666411 13588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:58.667716 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:58.669055 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:58.682307 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:58.682267 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:58.682346 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:58.683133 13588 server_base.cc:1034] running on GCE node
I20260928 15:02:58.683764 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:58.684727 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:58.685923 13588 hybrid_clock.cc:648] HybridClock initialized: now 1790607778685873 us; error 36 us; skew 500 ppm
I20260928 15:02:58.686237 13588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:58 dist-test-slave-dmhs krb5kdc[13348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:58.689328 13588 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:02:58.690470 13588 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:58.690594 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:58.690927 13588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:58.693773 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "21933354036047b898d391b3aa091f60"
format_stamp: "Formatted at 2026-09-28 15:02:58 on dist-test-slave-dmhs"
server_key: "afb0151a8519a72a384af360454504cd"
server_key_iv: "0d2300d125ea27581f23d2cb67ebdea3"
server_key_version: "encryptionkey@0"
I20260928 15:02:58.694657 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "21933354036047b898d391b3aa091f60"
format_stamp: "Formatted at 2026-09-28 15:02:58 on dist-test-slave-dmhs"
server_key: "afb0151a8519a72a384af360454504cd"
server_key_iv: "0d2300d125ea27581f23d2cb67ebdea3"
server_key_version: "encryptionkey@0"
I20260928 15:02:58.699592 13588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 15:02:58.703027 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:58.704227 13588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:02:58.704445 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "21933354036047b898d391b3aa091f60"
format_stamp: "Formatted at 2026-09-28 15:02:58 on dist-test-slave-dmhs"
server_key: "afb0151a8519a72a384af360454504cd"
server_key_iv: "0d2300d125ea27581f23d2cb67ebdea3"
server_key_version: "encryptionkey@0"
I20260928 15:02:58.704694 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:58.724239 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:58.729151 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:58.729409 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:58.730379 13588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:58.731592 13588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:58.731676 13588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:58.731751 13588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:58.731806 13588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:58.765158 13588 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:46011
I20260928 15:02:58.765209 13716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:46011 every 8 connection(s)
I20260928 15:02:58.766039 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:58.776103 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13588
I20260928 15:02:58.776309 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:58.776734 9267 external_mini_cluster.cc:1428] Setting key 859a3f30af338d001260d94a6f6f2ee7
Sep 28 15:02:58 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:58.786433 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:58.768654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:55115 (local address 127.9.12.254:40161)
0928 15:02:58.768836 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:58.768841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:58.769998 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 15:02:58.771845 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:58.771850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:58.771946 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:58.772056 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:58.773612 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.774334 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.775868 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.776399 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.776536 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:58.780558 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:58.780563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:58.780567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:58.780596 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:58.782991 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:58.783928 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:58.783932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:58.783934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:58.784069 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:58.784556 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:58.784563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:58.784567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:58.784940 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:58.785102 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:58.785727 (+ 625us) server_negotiation.cc:299] Negotiation successful
0928 15:02:58.785969 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:02:58.787377 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:58.769100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40161 (local address 127.9.12.194:55115)
0928 15:02:58.769842 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:58.769884 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:58.771588 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:58.772367 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:58.772375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:58.772882 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:58.773432 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:58.773449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.774497 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.774501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:58.775703 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:58.775715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.776623 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.776630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:58.776892 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:58.777590 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:58.777616 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:58.780347 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:58.783128 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:58.783139 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:58.783157 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:58.783804 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:58.784174 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:58.784181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:58.784187 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:58.784440 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:58.785063 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:58.785070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:58.785355 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 15:02:58.785576 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 15:02:58.785928 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":281,"threads_started":1}
I20260928 15:02:58.788971 13717 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40161
I20260928 15:02:58.789273 13717 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:58.790051 13717 heartbeater.cc:511] Master 127.9.12.254:40161 requested a full tablet report, sending...
I20260928 15:02:58.791543 13392 ts_manager.cc:194] Registered new tserver with Master: 21933354036047b898d391b3aa091f60 (127.9.12.194:46011)
I20260928 15:02:58.792383 13392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:55115
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:02:58.821668 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40161
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:59.136765 13722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:59.137032 13722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:59.137208 13722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:59.144234 13722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:59.144415 13722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:59.144611 13722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:02:59.154520 13722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:02:59.155776 13722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:59.157078 13722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:59.169986 13727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:59.169961 13730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:59.170048 13728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:59.171051 13722 server_base.cc:1034] running on GCE node
I20260928 15:02:59.171603 13722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:59.172526 13722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:59.173732 13722 hybrid_clock.cc:648] HybridClock initialized: now 1790607779173689 us; error 36 us; skew 500 ppm
I20260928 15:02:59.174054 13722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:02:59 dist-test-slave-dmhs krb5kdc[13348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:59.177150 13722 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:02:59.178325 13722 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:59.178483 13722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:59.178799 13722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:59.181734 13722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "0385e1ec81f94bda8097e930a5537c29"
format_stamp: "Formatted at 2026-09-28 15:02:59 on dist-test-slave-dmhs"
server_key: "00860f4abebeaef862b1eb5a2ce24a5e"
server_key_iv: "dd377a4a47fc2e0b9047a6f518747841"
server_key_version: "encryptionkey@0"
I20260928 15:02:59.182559 13722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "0385e1ec81f94bda8097e930a5537c29"
format_stamp: "Formatted at 2026-09-28 15:02:59 on dist-test-slave-dmhs"
server_key: "00860f4abebeaef862b1eb5a2ce24a5e"
server_key_iv: "dd377a4a47fc2e0b9047a6f518747841"
server_key_version: "encryptionkey@0"
I20260928 15:02:59.187598 13722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:02:59.190948 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:59.192212 13722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:02:59.192409 13722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "0385e1ec81f94bda8097e930a5537c29"
format_stamp: "Formatted at 2026-09-28 15:02:59 on dist-test-slave-dmhs"
server_key: "00860f4abebeaef862b1eb5a2ce24a5e"
server_key_iv: "dd377a4a47fc2e0b9047a6f518747841"
server_key_version: "encryptionkey@0"
I20260928 15:02:59.192621 13722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:59.211951 13722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:59.216351 13722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:59.216641 13722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:59.217617 13722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:59.218941 13722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:59.219071 13722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:59.219228 13722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:59.219316 13722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:59.252089 13722 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:45197
I20260928 15:02:59.252126 13849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:45197 every 8 connection(s)
I20260928 15:02:59.252975 13722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:59.259193 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13722
I20260928 15:02:59.259403 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:59.259872 9267 external_mini_cluster.cc:1428] Setting key 2aac2560949484d2489bc17006c86074
Sep 28 15:02:59 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:02:59.273156 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.255409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:36713 (local address 127.9.12.254:40161)
0928 15:02:59.255606 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.255611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.256819 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 15:02:59.258583 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.258589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.258704 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.258813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:59.260543 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.261662 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.263357 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.263765 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.263862 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.267077 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:59.267081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:59.267084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:59.267114 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:59.270021 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.270725 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.270730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.270732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.270851 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.271258 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.271262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.271268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.271541 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:59.271703 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.272570 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.272787 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:02:59.274443 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.255925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40161 (local address 127.9.12.195:36713)
0928 15:02:59.256665 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.256705 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:59.258313 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:59.259040 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:59.259049 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:59.259678 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:59.260348 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.260388 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.261845 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.261849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.263234 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.263246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.263942 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.263947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.264132 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.264798 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:59.264819 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:59.266858 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:59.270138 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:59.270144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:59.270156 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:59.270560 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:59.270959 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:59.270965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:59.270968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:59.271137 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:59.271677 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:59.271684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:59.272105 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 15:02:59.272470 (+ 365us) client_negotiation.cc:240] Negotiation successful
0928 15:02:59.273000 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":343,"threads_started":1}
I20260928 15:02:59.276379 13850 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40161
I20260928 15:02:59.276711 13850 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:59.277527 13850 heartbeater.cc:511] Master 127.9.12.254:40161 requested a full tablet report, sending...
I20260928 15:02:59.278925 13392 ts_manager.cc:194] Registered new tserver with Master: 0385e1ec81f94bda8097e930a5537c29 (127.9.12.195:45197)
I20260928 15:02:59.279716 13392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:36713
I20260928 15:02:59.288604 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:59.305235 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.293145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.9.12.254:40161)
0928 15:02:59.293303 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.293307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.293824 (+ 517us) server_negotiation.cc:407] Connection header received
0928 15:02:59.293969 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.293973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.294030 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.294110 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:59.295517 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.296223 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.297582 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.297988 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.298079 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.300194 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:59.300198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:59.300201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:59.300228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:59.303015 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.303558 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.303561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.303563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.303698 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.304043 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.304046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.304048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.304401 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:59.304586 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.304852 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.305002 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:02:59.309099 13392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:59.315915 13392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:59.318221 13582 heartbeater.cc:503] Master 127.9.12.254:40161 was elected leader, sending a full tablet report...
I20260928 15:02:59.350545 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.328126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:45197 (local address 127.0.0.1:59212)
0928 15:02:59.341968 (+ 13842us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.341983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:59.342100 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:59.342868 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:59.342874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:59.342902 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:59.343377 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.343393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.345562 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.345569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.347620 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.347633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.349518 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.349525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.349713 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.349778 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 15:02:59.349887 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:02:59.350093 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13707,"mutex_wait_us":47,"thread_start_us":12746,"threads_started":1}
I20260928 15:02:59.350716 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.327901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48420)
0928 15:02:59.341257 (+ 13356us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.341291 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:59.341494 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:59.342446 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:59.342453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:59.342505 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:59.343030 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.343047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.345130 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.345136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.347789 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.347804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.349834 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.349839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.350063 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.350096 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:02:59.350175 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:02:59.350308 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13109,"mutex_wait_us":87,"thread_start_us":13813,"threads_started":1}
I20260928 15:02:59.350859 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.328547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.9.12.195:45197)
0928 15:02:59.342108 (+ 13561us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.342119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.342151 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:02:59.342245 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.342254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.342463 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.343601 (+ 1138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:59.343885 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.345409 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.347789 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.349285 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.350047 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.350280 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.350356 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.350524 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13366,"thread_start_us":14051,"threads_started":1}
I20260928 15:02:59.351421 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.330035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:46011 (local address 127.0.0.1:45098)
0928 15:02:59.343063 (+ 13028us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.343086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:59.343203 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:59.343821 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:59.343826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:59.343851 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:59.344243 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.344252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.345837 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.345843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.348040 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.348049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.349792 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.349799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.350001 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.350070 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 15:02:59.350154 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:02:59.350302 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12880,"mutex_wait_us":12,"thread_start_us":13256,"threads_started":1}
I20260928 15:02:59.351584 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.330745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.9.12.194:46011)
0928 15:02:59.342550 (+ 11805us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.342559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.343340 (+ 781us) server_negotiation.cc:407] Connection header received
0928 15:02:59.343451 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.343460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.343648 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.343833 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:59.344455 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.345664 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.348185 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.349304 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.349434 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.349629 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.350497 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.350735 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11673,"thread_start_us":11568,"threads_started":1}
I20260928 15:02:59.351637 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.328056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.9.12.193:45367)
0928 15:02:59.340969 (+ 12913us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.340979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.341403 (+ 424us) server_negotiation.cc:407] Connection header received
0928 15:02:59.341711 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.341723 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.341980 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.342170 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:59.343234 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.344981 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.347942 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.349326 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.349455 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.349688 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.351128 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.351287 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12687,"thread_start_us":12923,"threads_started":1}
I20260928 15:02:59.355195 13517 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 (DEFAULT_TABLE table=test-table [id=7faa7d836f2744eb9909ae5501cbd642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:59.355232 13785 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 (DEFAULT_TABLE table=test-table [id=7faa7d836f2744eb9909ae5501cbd642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:59.355330 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 (DEFAULT_TABLE table=test-table [id=7faa7d836f2744eb9909ae5501cbd642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:59.356487 13517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:59.356583 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:59.356587 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:59.368232 13868 tablet_bootstrap.cc:492] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Bootstrap starting.
I20260928 15:02:59.371274 13869 tablet_bootstrap.cc:492] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Bootstrap starting.
I20260928 15:02:59.373009 13868 tablet_bootstrap.cc:654] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:59.374297 13868 log.cc:824] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:59.374611 13869 tablet_bootstrap.cc:654] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:59.375209 13870 tablet_bootstrap.cc:492] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Bootstrap starting.
I20260928 15:02:59.375964 13869 log.cc:824] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:59.378582 13868 tablet_bootstrap.cc:492] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: No bootstrap required, opened a new log
I20260928 15:02:59.378718 13870 tablet_bootstrap.cc:654] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:59.378913 13868 ts_tablet_manager.cc:1397] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 15:02:59.378985 13869 tablet_bootstrap.cc:492] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: No bootstrap required, opened a new log
I20260928 15:02:59.379236 13869 ts_tablet_manager.cc:1397] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:02:59.379877 13870 log.cc:824] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:59.383755 13870 tablet_bootstrap.cc:492] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: No bootstrap required, opened a new log
I20260928 15:02:59.383994 13870 ts_tablet_manager.cc:1397] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 15:02:59.386958 13868 raft_consensus.cc:348] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.387326 13868 raft_consensus.cc:374] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:59.387427 13868 raft_consensus.cc:729] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70e3ecdd2d74202951afdae68cc504d, State: Initialized, Role: FOLLOWER
I20260928 15:02:59.388056 13868 consensus_queue.cc:260] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.390017 13868 ts_tablet_manager.cc:1428] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 15:02:59.389972 13869 raft_consensus.cc:348] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.390342 13869 raft_consensus.cc:374] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:59.390425 13869 raft_consensus.cc:729] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0385e1ec81f94bda8097e930a5537c29, State: Initialized, Role: FOLLOWER
I20260928 15:02:59.391026 13869 consensus_queue.cc:260] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.392621 13850 heartbeater.cc:503] Master 127.9.12.254:40161 was elected leader, sending a full tablet report...
I20260928 15:02:59.394497 13870 raft_consensus.cc:348] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.394804 13869 ts_tablet_manager.cc:1428] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 15:02:59.394907 13870 raft_consensus.cc:374] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:59.394999 13870 raft_consensus.cc:729] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21933354036047b898d391b3aa091f60, State: Initialized, Role: FOLLOWER
I20260928 15:02:59.395543 13870 consensus_queue.cc:260] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.396642 13717 heartbeater.cc:503] Master 127.9.12.254:40161 was elected leader, sending a full tablet report...
I20260928 15:02:59.397320 13870 ts_tablet_manager.cc:1428] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260928 15:02:59.504601 13851 tablet.cc:2369] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:59.517906 13718 tablet.cc:2369] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:59.524967 13875 raft_consensus.cc:482] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:59.525280 13875 raft_consensus.cc:504] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.526925 13875 leader_election.cc:288] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c70e3ecdd2d74202951afdae68cc504d (127.9.12.193:45367), 21933354036047b898d391b3aa091f60 (127.9.12.194:46011)
I20260928 15:02:59.533576 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.527227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:46011 (local address 127.9.12.195:48089)
0928 15:02:59.527554 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.527578 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:59.527733 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:59.528168 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:59.528171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:59.528193 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:59.528513 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.528525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.529731 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.529736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.531524 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.531540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.533001 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.533005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.533134 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.533158 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:02:59.533232 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:02:59.533318 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"spinlock_wait_cycles":29568}
I20260928 15:02:59.533676 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.527471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:48089 (local address 127.9.12.194:46011)
0928 15:02:59.527666 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.527673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.527697 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:59.527927 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.527934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.528014 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.528139 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:59.528682 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.529592 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.531704 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.532857 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.533017 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.533103 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.533284 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.533413 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:02:59.534391 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.527397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:50731 (local address 127.9.12.193:45367)
0928 15:02:59.527601 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.527607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.528284 (+ 677us) server_negotiation.cc:407] Connection header received
0928 15:02:59.528483 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.528488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.528580 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.528678 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:59.529218 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.530056 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.532181 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.533431 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.533593 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.533658 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.534017 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.534152 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:02:59.534431 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.527313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.9.12.195:50731)
0928 15:02:59.528150 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.528173 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:59.528331 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:59.528693 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:59.528698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:59.528734 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:59.529079 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.529089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.530202 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.530206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.532030 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:59.532041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.533641 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.533647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:59.533847 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.533887 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:02:59.533999 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 15:02:59.534130 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":438,"threads_started":1}
I20260928 15:02:59.534830 13672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cf5c38d8942c79ca5c87cca96a1a3" candidate_uuid: "0385e1ec81f94bda8097e930a5537c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21933354036047b898d391b3aa091f60" is_pre_election: true
I20260928 15:02:59.535342 13672 raft_consensus.cc:2454] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0385e1ec81f94bda8097e930a5537c29 in term 0.
I20260928 15:02:59.535424 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cf5c38d8942c79ca5c87cca96a1a3" candidate_uuid: "0385e1ec81f94bda8097e930a5537c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70e3ecdd2d74202951afdae68cc504d" is_pre_election: true
I20260928 15:02:59.535990 13537 raft_consensus.cc:2454] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0385e1ec81f94bda8097e930a5537c29 in term 0.
I20260928 15:02:59.536023 13738 leader_election.cc:302] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0385e1ec81f94bda8097e930a5537c29, 21933354036047b898d391b3aa091f60; no voters:
I20260928 15:02:59.536441 13875 raft_consensus.cc:2781] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:59.536628 13875 raft_consensus.cc:482] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:59.536775 13875 raft_consensus.cc:3037] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:02:59.537623 13583 tablet.cc:2369] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:59.539386 13875 raft_consensus.cc:504] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.539922 13875 leader_election.cc:288] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [CANDIDATE]: Term 1 election: Requested vote from peers c70e3ecdd2d74202951afdae68cc504d (127.9.12.193:45367), 21933354036047b898d391b3aa091f60 (127.9.12.194:46011)
I20260928 15:02:59.540392 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cf5c38d8942c79ca5c87cca96a1a3" candidate_uuid: "0385e1ec81f94bda8097e930a5537c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70e3ecdd2d74202951afdae68cc504d"
I20260928 15:02:59.540467 13672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cf5c38d8942c79ca5c87cca96a1a3" candidate_uuid: "0385e1ec81f94bda8097e930a5537c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21933354036047b898d391b3aa091f60"
I20260928 15:02:59.540593 13537 raft_consensus.cc:3037] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:59.540709 13672 raft_consensus.cc:3037] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:59.542805 13537 raft_consensus.cc:2454] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0385e1ec81f94bda8097e930a5537c29 in term 1.
I20260928 15:02:59.542903 13672 raft_consensus.cc:2454] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0385e1ec81f94bda8097e930a5537c29 in term 1.
I20260928 15:02:59.543226 13740 leader_election.cc:302] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0385e1ec81f94bda8097e930a5537c29, c70e3ecdd2d74202951afdae68cc504d; no voters:
I20260928 15:02:59.543527 13875 raft_consensus.cc:2781] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:59.543987 13875 raft_consensus.cc:686] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0385e1ec81f94bda8097e930a5537c29, State: Running, Role: LEADER
I20260928 15:02:59.544523 13875 consensus_queue.cc:237] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } }
I20260928 15:02:59.549265 13392 catalog_manager.cc:5495] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0385e1ec81f94bda8097e930a5537c29 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "0385e1ec81f94bda8097e930a5537c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0385e1ec81f94bda8097e930a5537c29" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:59.589494 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.9.12.195:45197)
0928 15:02:59.584338 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.584344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.584442 (+ 98us) server_negotiation.cc:407] Connection header received
0928 15:02:59.584689 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.584694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.584780 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.584955 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:59.585343 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.586053 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.587376 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.587767 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.587896 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.588408 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:59.588915 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:59.589068 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.589148 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.589225 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:02:59.597440 13672 raft_consensus.cc:1261] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 1 FOLLOWER]: Refusing update from remote peer 0385e1ec81f94bda8097e930a5537c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:59.597430 13537 raft_consensus.cc:1261] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 1 FOLLOWER]: Refusing update from remote peer 0385e1ec81f94bda8097e930a5537c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:59.598384 13875 consensus_queue.cc:1035] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70e3ecdd2d74202951afdae68cc504d" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:59.598652 13878 consensus_queue.cc:1035] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21933354036047b898d391b3aa091f60" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:59.609901 13881 mvcc.cc:205] Tried to move back new op lower bound from 7334329465221062656 to 7334329465018261504. Current Snapshot: MvccSnapshot[applied={T|T < 7334329465221062656}]
I20260928 15:02:59.614802 13882 mvcc.cc:205] Tried to move back new op lower bound from 7334329465221062656 to 7334329465018261504. Current Snapshot: MvccSnapshot[applied={T|T < 7334329465221062656}]
I20260928 15:02:59.622546 13392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:59.622818 13392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:59.626389 13392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5793ea6ad5194196a9fe26e2ada10c58: Sending DeleteTablet for 3 replicas of tablet 4e2cf5c38d8942c79ca5c87cca96a1a3
I20260928 15:02:59.627432 13517 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:59 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48420
I20260928 15:02:59.627791 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:59 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:45098
I20260928 15:02:59.628145 13785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:59 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:59212
I20260928 15:02:59.629482 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13454
I20260928 15:02:59.643081 13891 tablet_replica.cc:330] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: stopping tablet replica
I20260928 15:02:59.643610 13891 raft_consensus.cc:2229] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:59.644162 13891 raft_consensus.cc:2258] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:59.646519 13891 ts_tablet_manager.cc:1905] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:59.647341 13892 tablet_replica.cc:330] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: stopping tablet replica
I20260928 15:02:59.647898 13892 raft_consensus.cc:2229] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:59.648671 13892 raft_consensus.cc:2258] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:59.649513 13890 tablet_replica.cc:330] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: stopping tablet replica
I20260928 15:02:59.651026 13892 ts_tablet_manager.cc:1905] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:02:59 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:02:59.654241 13890 raft_consensus.cc:2229] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:59.654594 13891 ts_tablet_manager.cc:1918] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:59.654768 13891 log.cc:1196] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/4e2cf5c38d8942c79ca5c87cca96a1a3
I20260928 15:02:59.654768 13890 raft_consensus.cc:2258] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:59.655306 13891 ts_tablet_manager.cc:1939] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 21933354036047b898d391b3aa091f60: Deleting consensus metadata
I20260928 15:02:59.658537 13377 catalog_manager.cc:4841] TS 21933354036047b898d391b3aa091f60 (127.9.12.194:46011): tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 (table test-table [id=7faa7d836f2744eb9909ae5501cbd642]) successfully deleted
I20260928 15:02:59.659061 13892 ts_tablet_manager.cc:1918] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:59.659225 13892 log.cc:1196] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/4e2cf5c38d8942c79ca5c87cca96a1a3
I20260928 15:02:59.659744 13892 ts_tablet_manager.cc:1939] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P 0385e1ec81f94bda8097e930a5537c29: Deleting consensus metadata
I20260928 15:02:59.661345 13378 catalog_manager.cc:4841] TS 0385e1ec81f94bda8097e930a5537c29 (127.9.12.195:45197): tablet 4e2cf5c38d8942c79ca5c87cca96a1a3 (table test-table [id=7faa7d836f2744eb9909ae5501cbd642]) successfully deleted
I20260928 15:02:59.661182 13890 ts_tablet_manager.cc:1905] T 4e2cf5c38d8942c79ca5c87cca96a1a3 P c70e3ecdd2d74202951afdae68cc504d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:59.662520 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.632113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.9.12.193:45367)
0928 15:02:59.632362 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.632371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.641595 (+ 9224us) server_negotiation.cc:407] Connection header received
0928 15:02:59.642143 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.642152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.642275 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.642367 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:59.644799 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.645918 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.647858 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.648525 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.648619 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.655252 (+ 6633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:59.655269 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:59.655305 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:59.655359 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:59.659225 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.659996 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.660004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.660007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.660191 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.660715 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.660720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.660723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.661151 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:59.661439 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.661911 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.662149 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:02:59.663230 13577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:59.772552 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13454
W20260928 15:02:59.798149 13379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.12.193:45367 (error 108)
I20260928 15:02:59.799196 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13588
I20260928 15:02:59.810096 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.809456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48452)
0928 15:02:59.809739 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.809897 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 15:02:59.810400 13379 catalog_manager.cc:4584] TS c70e3ecdd2d74202951afdae68cc504d (127.9.12.193:45367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e2cf5c38d8942c79ca5c87cca96a1a3: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Sep 28 15:02:59 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:02:59.823509 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.800429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.9.12.194:46011)
0928 15:02:59.800651 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.800659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.801021 (+ 362us) server_negotiation.cc:407] Connection header received
0928 15:02:59.801321 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.801327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.801455 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.801547 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:59.803656 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.805190 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.807465 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.808213 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.808373 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.815328 (+ 6955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:59.815355 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:59.815377 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:59.815444 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:59.819722 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.820885 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.820900 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.820906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.821125 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.821610 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.821614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.821618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.822167 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:59.822499 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.822849 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.823135 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 15:02:59.824404 13712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:02:59.864428 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.863757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48456)
0928 15:02:59.864052 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.864198 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 15:02:59.865959 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.865451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48470)
0928 15:02:59.865659 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.865783 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:02:59.927106 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.926430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48476)
0928 15:02:59.926737 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.926861 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 15:02:59.928579 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.928049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48488)
0928 15:02:59.928255 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:59.928381 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:02:59.933619 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13588
I20260928 15:02:59.951114 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13722
Sep 28 15:02:59 dist-test-slave-dmhs krb5kdc[13348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:02:59.969482 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.952040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.9.12.195:45197)
0928 15:02:59.952299 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.952307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.952401 (+ 94us) server_negotiation.cc:407] Connection header received
0928 15:02:59.952652 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.952656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.952742 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.952861 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:59.954294 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.955127 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.956590 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.957060 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.957230 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.962084 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:59.962103 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:59.962119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:59.962174 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:59.965858 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.966784 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.966792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.966795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.967005 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.967493 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.967497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.967499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.967893 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:59.968294 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.968786 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.969073 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 15:02:59.970309 13845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:00.009619 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.009077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48492)
0928 15:03:00.009321 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:00.009418 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 15:03:00.010690 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.010303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48494)
0928 15:03:00.010501 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:00.010581 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 15:03:00.099538 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13722
I20260928 15:03:00.120345 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13361
I20260928 15:03:00.121685 13423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:00.235181 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13361
I20260928 15:03:00.235252 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.234706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45367 (local address 127.0.0.1:48510)
0928 15:03:00.234909 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:00.235024 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
2026-09-28T15:03:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4210 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T15:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:00Z Disabled control of system clock
I20260928 15:03:00.310050 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34705
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:34705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:03:00.657280 13902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:00.657719 13902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:00.657873 13902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:00.670375 13902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:00.670539 13902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:00.670610 13902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:00.670694 13902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:00.670761 13902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:00.682577 13902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:34705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:00.684010 13902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:00.685940 13902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:00.701395 13908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:00.701469 13907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:00.701493 13910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:00.702368 13902 server_base.cc:1034] running on GCE node
I20260928 15:03:00.703281 13902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:00.704888 13902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:00.706154 13902 hybrid_clock.cc:648] HybridClock initialized: now 1790607780706109 us; error 55 us; skew 500 ppm
I20260928 15:03:00.706552 13902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:00.709089 13902 webserver.cc:460] Webserver started at http://127.9.12.254:33733/ using document root <none> and password file <none>
I20260928 15:03:00.709834 13902 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:00.709977 13902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:00.710383 13902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:00.713526 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "c64c2bbcd77749cc907758c8b6ef2422"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-dmhs"
server_key: "196e0483a26c7bb2d2f1eea9e03c843a"
server_key_iv: "09e83d94c83e1cee0e4f9fe6161ecd8a"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.714383 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "c64c2bbcd77749cc907758c8b6ef2422"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-dmhs"
server_key: "196e0483a26c7bb2d2f1eea9e03c843a"
server_key_iv: "09e83d94c83e1cee0e4f9fe6161ecd8a"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.719926 13902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:03:00.723938 13916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.725651 13902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:00.725898 13902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "c64c2bbcd77749cc907758c8b6ef2422"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-dmhs"
server_key: "196e0483a26c7bb2d2f1eea9e03c843a"
server_key_iv: "09e83d94c83e1cee0e4f9fe6161ecd8a"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.726187 13902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:00.733126 13902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:00.734095 13902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:00.734448 13902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:00.756011 13902 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:34705
I20260928 15:03:00.756021 13968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:34705 every 8 connection(s)
I20260928 15:03:00.757716 13902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:00.761796 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13902
I20260928 15:03:00.762007 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:00.762794 13969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:00.762879 9267 external_mini_cluster.cc:1428] Setting key 33442ea988465198f8dbc483ca16ae10
I20260928 15:03:00.773211 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.765330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.9.12.254:34705)
0928 15:03:00.766641 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.766654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.766713 (+ 59us) server_negotiation.cc:407] Connection header received
0928 15:03:00.768717 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.768751 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.769339 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.769824 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.771223 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.771230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.771245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:00.771263 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.771335 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:00.771360 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.771571 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.771687 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.772008 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":448,"threads_started":1}
I20260928 15:03:00.778308 13969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422: Bootstrap starting.
I20260928 15:03:00.781879 13969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:00.783115 13969 log.cc:824] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:00.786533 13969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422: No bootstrap required, opened a new log
I20260928 15:03:00.794920 13969 raft_consensus.cc:348] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64c2bbcd77749cc907758c8b6ef2422" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34705 } }
I20260928 15:03:00.795437 13969 raft_consensus.cc:374] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:00.795589 13969 raft_consensus.cc:729] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64c2bbcd77749cc907758c8b6ef2422, State: Initialized, Role: FOLLOWER
I20260928 15:03:00.796439 13969 consensus_queue.cc:260] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64c2bbcd77749cc907758c8b6ef2422" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34705 } }
I20260928 15:03:00.796767 13969 raft_consensus.cc:388] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:00.796937 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:00.797093 13969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:00.799638 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64c2bbcd77749cc907758c8b6ef2422" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34705 } }
I20260928 15:03:00.800388 13969 leader_election.cc:302] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c64c2bbcd77749cc907758c8b6ef2422; no voters:
I20260928 15:03:00.801095 13969 leader_election.cc:288] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:00.801272 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:00.802088 13974 raft_consensus.cc:686] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [term 1 LEADER]: Becoming Leader. State: Replica: c64c2bbcd77749cc907758c8b6ef2422, State: Running, Role: LEADER
I20260928 15:03:00.802713 13974 consensus_queue.cc:237] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64c2bbcd77749cc907758c8b6ef2422" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34705 } }
I20260928 15:03:00.803416 13969 sys_catalog.cc:564] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:00.806313 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c64c2bbcd77749cc907758c8b6ef2422. Latest consensus state: current_term: 1 leader_uuid: "c64c2bbcd77749cc907758c8b6ef2422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64c2bbcd77749cc907758c8b6ef2422" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34705 } } }
I20260928 15:03:00.806797 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:00.807098 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c64c2bbcd77749cc907758c8b6ef2422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64c2bbcd77749cc907758c8b6ef2422" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34705 } } }
I20260928 15:03:00.807499 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:00.809080 13981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:00.816183 13981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:00.826898 13981 catalog_manager.cc:1347] Generated new cluster ID: 3d4ba5db1344419faf91a92f231afb64
I20260928 15:03:00.827064 13981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:00.863008 13981 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:00.863536 13981 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:00.874760 13981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422: Generated new TSK 0
I20260928 15:03:00.876039 13981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:00.896055 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34705
--builtin_ntp_servers=127.9.12.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:03:01.273348 13993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:01.273669 13993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:01.273846 13993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:01.283120 13993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:01.283298 13993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:01.283495 13993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:01.293768 13993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:01.295249 13993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:01.297302 13993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:01.312177 14001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:01.312280 13998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:01.312237 13999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:01.313213 13993 server_base.cc:1034] running on GCE node
I20260928 15:03:01.313905 13993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:01.314968 13993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:01.316188 13993 hybrid_clock.cc:648] HybridClock initialized: now 1790607781316139 us; error 43 us; skew 500 ppm
I20260928 15:03:01.316571 13993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:01.328722 13993 webserver.cc:460] Webserver started at http://127.9.12.193:34781/ using document root <none> and password file <none>
I20260928 15:03:01.329573 13993 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:01.329698 13993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:01.330094 13993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:01.333509 13993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "9a153c21841c4bb4b544b62cc991fd6c"
format_stamp: "Formatted at 2026-09-28 15:03:01 on dist-test-slave-dmhs"
server_key: "e7fa426eec89d5fd71692569ebf8553d"
server_key_iv: "6b574deed169c1696ce981464b0d46ff"
server_key_version: "encryptionkey@0"
I20260928 15:03:01.334456 13993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "9a153c21841c4bb4b544b62cc991fd6c"
format_stamp: "Formatted at 2026-09-28 15:03:01 on dist-test-slave-dmhs"
server_key: "e7fa426eec89d5fd71692569ebf8553d"
server_key_iv: "6b574deed169c1696ce981464b0d46ff"
server_key_version: "encryptionkey@0"
I20260928 15:03:01.340466 13993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:03:01.344148 14007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:01.345772 13993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:01.346017 13993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "9a153c21841c4bb4b544b62cc991fd6c"
format_stamp: "Formatted at 2026-09-28 15:03:01 on dist-test-slave-dmhs"
server_key: "e7fa426eec89d5fd71692569ebf8553d"
server_key_iv: "6b574deed169c1696ce981464b0d46ff"
server_key_version: "encryptionkey@0"
I20260928 15:03:01.346297 13993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:01.352522 13993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:01.353387 13993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:01.353675 13993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:01.354643 13993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:01.356096 13993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:01.356213 13993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:01.356382 13993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:01.356487 13993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:01.390811 13993 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:36115
I20260928 15:03:01.390866 14120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:36115 every 8 connection(s)
I20260928 15:03:01.392216 13993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:01.401031 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:01.394886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42047 (local address 127.9.12.254:34705)
0928 15:03:01.395334 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:01.395339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:01.396714 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 15:03:01.398677 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:01.398685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:01.398768 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:01.398937 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:01.400348 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:01.400358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:01.400361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:01.400375 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:01.400413 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:01.400424 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:01.400530 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:01.400737 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 15:03:01.400791 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260928 15:03:01.401928 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:01.395518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34705 (local address 127.9.12.193:42047)
0928 15:03:01.396544 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:01.396586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:01.398293 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:01.399220 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:01.399229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:01.399324 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:01.400047 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:01.400050 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:01.400087 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:01.400170 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:01.400518 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:01.400525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:01.400530 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:01.400743 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 15:03:01.400936 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":309,"threads_started":1}
I20260928 15:03:01.402379 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 13993
I20260928 15:03:01.402572 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:01.403360 9267 external_mini_cluster.cc:1428] Setting key cdd06844c6a3ffd75b430f43c1d27f17
I20260928 15:03:01.403622 14121 heartbeater.cc:347] Connected to a master server at 127.9.12.254:34705
I20260928 15:03:01.403964 14121 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:01.404650 14121 heartbeater.cc:511] Master 127.9.12.254:34705 requested a full tablet report, sending...
I20260928 15:03:01.406301 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34705
--builtin_ntp_servers=127.9.12.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 15:03:01.406749 13933 ts_manager.cc:194] Registered new tserver with Master: 9a153c21841c4bb4b544b62cc991fd6c (127.9.12.193:36115)
W20260928 15:03:01.725910 14125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:01.726199 14125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:01.726383 14125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:01.733944 14125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:01.734055 14125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:01.734258 14125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:01.743716 14125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:01.745074 14125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:01.746840 14125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:01.760054 14130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:01.760587 14131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:01.760660 14133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:01.761186 14125 server_base.cc:1034] running on GCE node
I20260928 15:03:01.761888 14125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:01.763077 14125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:01.764335 14125 hybrid_clock.cc:648] HybridClock initialized: now 1790607781764267 us; error 56 us; skew 500 ppm
I20260928 15:03:01.764701 14125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:01.767473 14125 webserver.cc:460] Webserver started at http://127.9.12.194:35393/ using document root <none> and password file <none>
I20260928 15:03:01.768393 14125 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:01.768505 14125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:01.768857 14125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:01.772812 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "12f3c6e06fac4be1b2865e9a8f32c45b"
format_stamp: "Formatted at 2026-09-28 15:03:01 on dist-test-slave-dmhs"
server_key: "fd3fc96d36a41e78215855adf486440a"
server_key_iv: "338ee8fdc7cabafef9a157653414d649"
server_key_version: "encryptionkey@0"
I20260928 15:03:01.773830 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "12f3c6e06fac4be1b2865e9a8f32c45b"
format_stamp: "Formatted at 2026-09-28 15:03:01 on dist-test-slave-dmhs"
server_key: "fd3fc96d36a41e78215855adf486440a"
server_key_iv: "338ee8fdc7cabafef9a157653414d649"
server_key_version: "encryptionkey@0"
I20260928 15:03:01.779924 14125 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 15:03:01.783448 14139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:01.784828 14125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:01.785006 14125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "12f3c6e06fac4be1b2865e9a8f32c45b"
format_stamp: "Formatted at 2026-09-28 15:03:01 on dist-test-slave-dmhs"
server_key: "fd3fc96d36a41e78215855adf486440a"
server_key_iv: "338ee8fdc7cabafef9a157653414d649"
server_key_version: "encryptionkey@0"
I20260928 15:03:01.785243 14125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:01.790962 14125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:01.791790 14125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:01.792080 14125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:01.793008 14125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:01.794340 14125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:01.794618 14125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:01.794880 14125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:01.795006 14125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:01.828500 14125 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:38217
I20260928 15:03:01.828594 14252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:38217 every 8 connection(s)
I20260928 15:03:01.829769 14125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:01.836701 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14125
I20260928 15:03:01.836890 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:01.837390 9267 external_mini_cluster.cc:1428] Setting key d715e3471c8e34520b727f87deac6e20
I20260928 15:03:01.838020 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:01.832421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:35335 (local address 127.9.12.254:34705)
0928 15:03:01.832637 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:01.832642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:01.833971 (+ 1329us) server_negotiation.cc:407] Connection header received
0928 15:03:01.835601 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:01.835605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:01.835657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:01.835745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:01.837269 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:01.837274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:01.837279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:01.837305 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:01.837337 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:01.837348 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:01.837484 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:01.837687 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 15:03:01.837758 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:03:01.839169 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:01.832918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34705 (local address 127.9.12.194:35335)
0928 15:03:01.833819 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:01.833855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:01.835360 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:01.835941 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:01.835948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:01.836035 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:01.836894 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:01.836897 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:01.836974 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:01.837092 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:01.837450 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:01.837463 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:01.837471 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:03:01.837768 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 15:03:01.838020 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260928 15:03:01.840523 14253 heartbeater.cc:347] Connected to a master server at 127.9.12.254:34705
I20260928 15:03:01.840534 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34705
--builtin_ntp_servers=127.9.12.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 15:03:01.840866 14253 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:01.841588 14253 heartbeater.cc:511] Master 127.9.12.254:34705 requested a full tablet report, sending...
I20260928 15:03:01.843318 13933 ts_manager.cc:194] Registered new tserver with Master: 12f3c6e06fac4be1b2865e9a8f32c45b (127.9.12.194:38217)
W20260928 15:03:02.159783 14256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:02.160036 14256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:02.160130 14256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:02.167150 14256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:02.167243 14256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:02.167346 14256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:02.176152 14256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:34705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:02.177407 14256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:02.179032 14256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:02.191939 14262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:02.192018 14261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:02.192055 14264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:02.192847 14256 server_base.cc:1034] running on GCE node
I20260928 15:03:02.193437 14256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:02.194346 14256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:02.195575 14256 hybrid_clock.cc:648] HybridClock initialized: now 1790607782195539 us; error 55 us; skew 500 ppm
I20260928 15:03:02.195921 14256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:02.198207 14256 webserver.cc:460] Webserver started at http://127.9.12.195:36325/ using document root <none> and password file <none>
I20260928 15:03:02.198884 14256 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:02.199003 14256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:02.199301 14256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:02.201951 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "f2fd1d1c09e2497eabbdf70444dedf7b"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-dmhs"
server_key: "66ee0d872b05f316b57b6aeafc8d5e71"
server_key_iv: "e8b9f7486ffb1d81f1b60b089cc88953"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.202710 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "f2fd1d1c09e2497eabbdf70444dedf7b"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-dmhs"
server_key: "66ee0d872b05f316b57b6aeafc8d5e71"
server_key_iv: "e8b9f7486ffb1d81f1b60b089cc88953"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.207680 14256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:03:02.211125 14270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.212576 14256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:02.212743 14256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "f2fd1d1c09e2497eabbdf70444dedf7b"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-dmhs"
server_key: "66ee0d872b05f316b57b6aeafc8d5e71"
server_key_iv: "e8b9f7486ffb1d81f1b60b089cc88953"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.212984 14256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:02.219087 14256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:02.219856 14256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:02.220139 14256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:02.221095 14256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:02.222477 14256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:02.222602 14256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.222741 14256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:02.222831 14256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.254933 14256 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:40173
I20260928 15:03:02.254980 14383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:40173 every 8 connection(s)
I20260928 15:03:02.256274 14256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:02.261133 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14256
I20260928 15:03:02.261301 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:02.261731 9267 external_mini_cluster.cc:1428] Setting key 4cc427ad012fd93c9f5140c0d6a7745b
I20260928 15:03:02.264499 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.258711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:51733 (local address 127.9.12.254:34705)
0928 15:03:02.258886 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.258890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.260156 (+ 1266us) server_negotiation.cc:407] Connection header received
0928 15:03:02.262003 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.262009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.262089 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.262200 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.263724 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.263730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.263734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.263754 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.263798 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.263813 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.263933 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.264106 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.264171 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:03:02.265369 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.259127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34705 (local address 127.9.12.195:51733)
0928 15:03:02.259995 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.260035 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.261712 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.262437 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.262449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.262597 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.263352 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:02.263355 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.263392 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:02.263528 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.263913 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.263921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.263926 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.264136 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.264415 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":272,"threads_started":1}
I20260928 15:03:02.266613 14384 heartbeater.cc:347] Connected to a master server at 127.9.12.254:34705
I20260928 15:03:02.266860 14384 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:02.267424 14384 heartbeater.cc:511] Master 127.9.12.254:34705 requested a full tablet report, sending...
I20260928 15:03:02.268718 13933 ts_manager.cc:194] Registered new tserver with Master: f2fd1d1c09e2497eabbdf70444dedf7b (127.9.12.195:40173)
I20260928 15:03:02.277184 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:02.284379 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.281518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.9.12.254:34705)
0928 15:03:02.281721 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.281729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.282174 (+ 445us) server_negotiation.cc:407] Connection header received
0928 15:03:02.282382 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.282394 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.282448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.282551 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.283518 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.283523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.283528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.283554 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.283607 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.283624 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.283757 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.283905 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.283996 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:03:02.289129 13933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:02.296115 13933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:02.324455 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.308043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.9.12.193:36115)
0928 15:03:02.320754 (+ 12711us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.320760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.320783 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:03:02.320841 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.320876 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.321118 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.321281 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.323418 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.323432 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.323454 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.323488 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.323567 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.323593 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.323745 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.323960 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.324067 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12560,"thread_start_us":12424,"threads_started":1}
I20260928 15:03:02.324576 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.307979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51478)
0928 15:03:02.319358 (+ 11379us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.319410 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.319531 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.322281 (+ 2750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.322287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.322357 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.323111 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:02.323114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.323137 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:02.323206 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.323721 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.323736 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.323748 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.323965 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.324082 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11131,"thread_start_us":10963,"threads_started":1}
I20260928 15:03:02.325464 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.308049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:40173 (local address 127.0.0.1:35748)
0928 15:03:02.320400 (+ 12351us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.320423 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.320564 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.322961 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.322965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.322989 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.324338 (+ 1349us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:02.324344 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.324363 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:02.324406 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.324906 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.324913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.324918 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.325040 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.325082 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12197,"thread_start_us":12079,"threads_started":1}
I20260928 15:03:02.326648 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.308694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.9.12.195:40173)
0928 15:03:02.318960 (+ 10266us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.318975 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.320705 (+ 1730us) server_negotiation.cc:407] Connection header received
0928 15:03:02.320812 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.320824 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.321111 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.321290 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.324616 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.324626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.324642 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.324664 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.324732 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.324754 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.326046 (+ 1292us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.326113 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.326225 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10021,"thread_start_us":9815,"threads_started":1}
I20260928 15:03:02.327579 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.319837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:38217 (local address 127.0.0.1:37710)
0928 15:03:02.324048 (+ 4211us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.324065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.324163 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.325640 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.325644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.325664 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.326359 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:02.326362 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.326377 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:02.326416 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.326970 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.326977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.326983 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.327123 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.327169 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4095,"thread_start_us":3916,"threads_started":1}
I20260928 15:03:02.328084 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.320143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.9.12.194:38217)
0928 15:03:02.323547 (+ 3404us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.323553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.324165 (+ 612us) server_negotiation.cc:407] Connection header received
0928 15:03:02.324309 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.324318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.324561 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.324705 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.326584 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.326593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.326607 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.326629 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.326693 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.326723 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.326906 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.327765 (+ 859us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.327854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3215,"thread_start_us":2674,"threads_started":1}
I20260928 15:03:02.329953 14056 tablet_service.cc:1467] Processing CreateTablet for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 (DEFAULT_TABLE table=test-table [id=eaa1941aef6b466fbd6921caabb33542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:02.330310 14319 tablet_service.cc:1467] Processing CreateTablet for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 (DEFAULT_TABLE table=test-table [id=eaa1941aef6b466fbd6921caabb33542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:02.331552 14056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:02.331545 14319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:02.331259 14187 tablet_service.cc:1467] Processing CreateTablet for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 (DEFAULT_TABLE table=test-table [id=eaa1941aef6b466fbd6921caabb33542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:02.332558 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:02.346967 14404 tablet_bootstrap.cc:492] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Bootstrap starting.
I20260928 15:03:02.350008 14402 tablet_bootstrap.cc:492] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Bootstrap starting.
I20260928 15:03:02.351483 14404 tablet_bootstrap.cc:654] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:02.352596 14404 log.cc:824] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:02.353657 14403 tablet_bootstrap.cc:492] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Bootstrap starting.
I20260928 15:03:02.354107 14402 tablet_bootstrap.cc:654] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:02.355046 14402 log.cc:824] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:02.356630 14403 tablet_bootstrap.cc:654] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:02.357549 14403 log.cc:824] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:02.357779 14402 tablet_bootstrap.cc:492] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: No bootstrap required, opened a new log
I20260928 15:03:02.358075 14402 ts_tablet_manager.cc:1397] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 15:03:02.359980 14404 tablet_bootstrap.cc:492] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: No bootstrap required, opened a new log
I20260928 15:03:02.360337 14404 ts_tablet_manager.cc:1397] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260928 15:03:02.360596 14403 tablet_bootstrap.cc:492] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: No bootstrap required, opened a new log
I20260928 15:03:02.360877 14403 ts_tablet_manager.cc:1397] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260928 15:03:02.365789 14402 raft_consensus.cc:348] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.366163 14402 raft_consensus.cc:374] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:02.366314 14402 raft_consensus.cc:729] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f3c6e06fac4be1b2865e9a8f32c45b, State: Initialized, Role: FOLLOWER
I20260928 15:03:02.366964 14402 consensus_queue.cc:260] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.368005 14253 heartbeater.cc:503] Master 127.9.12.254:34705 was elected leader, sending a full tablet report...
I20260928 15:03:02.369217 14402 ts_tablet_manager.cc:1428] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260928 15:03:02.371388 14404 raft_consensus.cc:348] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.371388 14403 raft_consensus.cc:348] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.371816 14403 raft_consensus.cc:374] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:02.371901 14404 raft_consensus.cc:374] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:02.371964 14403 raft_consensus.cc:729] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fd1d1c09e2497eabbdf70444dedf7b, State: Initialized, Role: FOLLOWER
I20260928 15:03:02.372004 14404 raft_consensus.cc:729] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a153c21841c4bb4b544b62cc991fd6c, State: Initialized, Role: FOLLOWER
I20260928 15:03:02.372624 14403 consensus_queue.cc:260] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.372622 14404 consensus_queue.cc:260] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.373741 14121 heartbeater.cc:503] Master 127.9.12.254:34705 was elected leader, sending a full tablet report...
I20260928 15:03:02.374663 14404 ts_tablet_manager.cc:1428] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 15:03:02.375681 14384 heartbeater.cc:503] Master 127.9.12.254:34705 was elected leader, sending a full tablet report...
I20260928 15:03:02.376322 14403 ts_tablet_manager.cc:1428] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260928 15:03:02.394423 14122 tablet.cc:2369] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:02.440181 14409 raft_consensus.cc:482] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:02.440548 14409 raft_consensus.cc:504] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.442325 14409 leader_election.cc:288] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fd1d1c09e2497eabbdf70444dedf7b (127.9.12.195:40173), 12f3c6e06fac4be1b2865e9a8f32c45b (127.9.12.194:38217)
I20260928 15:03:02.446328 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.442667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:59867 (local address 127.9.12.195:40173)
0928 15:03:02.442865 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.442873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.444109 (+ 1236us) server_negotiation.cc:407] Connection header received
0928 15:03:02.444304 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.444309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.444370 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.444493 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.445600 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.445605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.445609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.445630 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.445694 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.445711 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.445806 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.446027 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.446095 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:03:02.446522 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.442730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:38217 (local address 127.9.12.193:51619)
0928 15:03:02.443412 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.443442 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.443702 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.444317 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.444322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.444347 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.445135 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:02.445137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.445152 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:02.445188 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.445575 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.445580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.445585 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.445780 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.445893 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":571,"threads_started":1}
I20260928 15:03:02.447305 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.443014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:40173 (local address 127.9.12.193:59867)
0928 15:03:02.443995 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.444010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.444135 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.444529 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.444532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.444568 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.445327 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:02.445329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.445341 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:02.445389 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.445889 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.445894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.445898 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.446799 (+ 901us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.446853 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":653,"threads_started":1}
I20260928 15:03:02.447623 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.442975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51619 (local address 127.9.12.194:38217)
0928 15:03:02.443181 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.443186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.443537 (+ 351us) server_negotiation.cc:407] Connection header received
0928 15:03:02.443893 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.443900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.443966 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.444040 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.445355 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.445360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.445363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.445379 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.445434 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.445450 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.445583 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.447131 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.447224 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:03:02.448961 14207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d747cbfa49f419e8e7c8a8cd1c56ce2" candidate_uuid: "9a153c21841c4bb4b544b62cc991fd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" is_pre_election: true
I20260928 15:03:02.449575 14207 raft_consensus.cc:2454] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a153c21841c4bb4b544b62cc991fd6c in term 0.
I20260928 15:03:02.449937 14339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d747cbfa49f419e8e7c8a8cd1c56ce2" candidate_uuid: "9a153c21841c4bb4b544b62cc991fd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" is_pre_election: true
I20260928 15:03:02.450804 14339 raft_consensus.cc:2454] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a153c21841c4bb4b544b62cc991fd6c in term 0.
I20260928 15:03:02.451514 14009 leader_election.cc:302] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f3c6e06fac4be1b2865e9a8f32c45b, 9a153c21841c4bb4b544b62cc991fd6c; no voters:
I20260928 15:03:02.451951 14409 raft_consensus.cc:2781] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:02.452111 14409 raft_consensus.cc:482] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:02.452222 14409 raft_consensus.cc:3037] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:02.455974 14409 raft_consensus.cc:504] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.456977 14409 leader_election.cc:288] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [CANDIDATE]: Term 1 election: Requested vote from peers f2fd1d1c09e2497eabbdf70444dedf7b (127.9.12.195:40173), 12f3c6e06fac4be1b2865e9a8f32c45b (127.9.12.194:38217)
I20260928 15:03:02.457458 14339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d747cbfa49f419e8e7c8a8cd1c56ce2" candidate_uuid: "9a153c21841c4bb4b544b62cc991fd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b"
I20260928 15:03:02.457684 14339 raft_consensus.cc:3037] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:02.458403 14207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d747cbfa49f419e8e7c8a8cd1c56ce2" candidate_uuid: "9a153c21841c4bb4b544b62cc991fd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b"
I20260928 15:03:02.458604 14207 raft_consensus.cc:3037] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:02.461097 14339 raft_consensus.cc:2454] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a153c21841c4bb4b544b62cc991fd6c in term 1.
I20260928 15:03:02.461716 14207 raft_consensus.cc:2454] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a153c21841c4bb4b544b62cc991fd6c in term 1.
I20260928 15:03:02.462105 14009 leader_election.cc:302] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f3c6e06fac4be1b2865e9a8f32c45b, 9a153c21841c4bb4b544b62cc991fd6c; no voters:
I20260928 15:03:02.462399 14409 raft_consensus.cc:2781] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:02.462833 14409 raft_consensus.cc:686] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 1 LEADER]: Becoming Leader. State: Replica: 9a153c21841c4bb4b544b62cc991fd6c, State: Running, Role: LEADER
I20260928 15:03:02.463402 14409 consensus_queue.cc:237] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } }
I20260928 15:03:02.468900 13932 catalog_manager.cc:5495] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a153c21841c4bb4b544b62cc991fd6c (127.9.12.193). New cstate: current_term: 1 leader_uuid: "9a153c21841c4bb4b544b62cc991fd6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a153c21841c4bb4b544b62cc991fd6c" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:02.483807 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.481645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.9.12.193:36115)
0928 15:03:02.481845 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.481851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.481960 (+ 109us) server_negotiation.cc:407] Connection header received
0928 15:03:02.482195 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.482199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.482274 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.482379 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.483190 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.483196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.483201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.483218 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.483263 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.483275 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.483389 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.483478 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.483538 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:03:02.492027 14207 raft_consensus.cc:1261] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 1 FOLLOWER]: Refusing update from remote peer 9a153c21841c4bb4b544b62cc991fd6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:02.492029 14339 raft_consensus.cc:1261] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 1 FOLLOWER]: Refusing update from remote peer 9a153c21841c4bb4b544b62cc991fd6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:02.493073 14413 consensus_queue.cc:1035] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3c6e06fac4be1b2865e9a8f32c45b" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:02.493434 14409 consensus_queue.cc:1035] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fd1d1c09e2497eabbdf70444dedf7b" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:02.521030 13932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:02.521322 13932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:02.525457 13932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c64c2bbcd77749cc907758c8b6ef2422: Sending DeleteTablet for 3 replicas of tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2
I20260928 15:03:02.526582 14056 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:02 UTC) from {username='slave'} at 127.0.0.1:51478
I20260928 15:03:02.526566 14319 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:02 UTC) from {username='slave'} at 127.0.0.1:35748
I20260928 15:03:02.527654 14187 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:02 UTC) from {username='slave'} at 127.0.0.1:37710
I20260928 15:03:02.528215 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13993
I20260928 15:03:02.543776 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.530192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.9.12.193:36115)
0928 15:03:02.530426 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.530435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.540721 (+ 10286us) server_negotiation.cc:407] Connection header received
0928 15:03:02.541254 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.541263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.541347 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.541435 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.542844 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.542853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.542860 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.542886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.542939 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.542960 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.543072 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.543346 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.543416 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:03:02.545285 14116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:02.546401 14428 tablet_replica.cc:330] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: stopping tablet replica
I20260928 15:03:02.547075 14428 raft_consensus.cc:2229] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:02.547770 14428 raft_consensus.cc:2258] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:02.549139 14426 tablet_replica.cc:330] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: stopping tablet replica
I20260928 15:03:02.549530 14426 raft_consensus.cc:2229] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:02.550024 14426 raft_consensus.cc:2258] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:02.550705 14428 ts_tablet_manager.cc:1905] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:02.551955 14426 ts_tablet_manager.cc:1905] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:02.558722 14428 ts_tablet_manager.cc:1918] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:02.558923 14428 log.cc:1196] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/4d747cbfa49f419e8e7c8a8cd1c56ce2
I20260928 15:03:02.559036 14426 ts_tablet_manager.cc:1918] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:02.559190 14426 log.cc:1196] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/4d747cbfa49f419e8e7c8a8cd1c56ce2
I20260928 15:03:02.559494 14428 ts_tablet_manager.cc:1939] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 12f3c6e06fac4be1b2865e9a8f32c45b: Deleting consensus metadata
I20260928 15:03:02.559702 14426 ts_tablet_manager.cc:1939] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P f2fd1d1c09e2497eabbdf70444dedf7b: Deleting consensus metadata
I20260928 15:03:02.561515 13918 catalog_manager.cc:4841] TS 12f3c6e06fac4be1b2865e9a8f32c45b (127.9.12.194:38217): tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 (table test-table [id=eaa1941aef6b466fbd6921caabb33542]) successfully deleted
I20260928 15:03:02.561488 13920 catalog_manager.cc:4841] TS f2fd1d1c09e2497eabbdf70444dedf7b (127.9.12.195:40173): tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2 (table test-table [id=eaa1941aef6b466fbd6921caabb33542]) successfully deleted
I20260928 15:03:02.652099 14427 tablet_replica.cc:330] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: stopping tablet replica
I20260928 15:03:02.652598 14427 raft_consensus.cc:2229] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:02.653185 14427 raft_consensus.cc:2258] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:02.655016 14427 ts_tablet_manager.cc:1905] T 4d747cbfa49f419e8e7c8a8cd1c56ce2 P 9a153c21841c4bb4b544b62cc991fd6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:02.655282 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13993
I20260928 15:03:02.672848 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14125
W20260928 15:03:02.672952 13918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.12.193:36115 (error 108)
I20260928 15:03:02.675029 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.674522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51490)
0928 15:03:02.674676 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.674826 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:03:02.675261 13918 catalog_manager.cc:4584] TS 9a153c21841c4bb4b544b62cc991fd6c (127.9.12.193:36115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d747cbfa49f419e8e7c8a8cd1c56ce2: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
I20260928 15:03:02.676050 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.673715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.9.12.194:38217)
0928 15:03:02.673913 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.673920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.674039 (+ 119us) server_negotiation.cc:407] Connection header received
0928 15:03:02.674188 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.674193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.674314 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.674427 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.675550 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.675554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.675557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.675571 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.675600 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.675612 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.675710 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.675813 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.675856 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:03:02.676720 14248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:02.728964 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.728512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51500)
0928 15:03:02.728722 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.728813 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 15:03:02.730001 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.729599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51516)
0928 15:03:02.729755 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.729832 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 15:03:02.765362 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14125
I20260928 15:03:02.784605 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14256
I20260928 15:03:02.787618 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.785375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.9.12.195:40173)
0928 15:03:02.785568 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.785574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.785595 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:03:02.785809 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.785813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.785893 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.786000 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.786975 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.786979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.786983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:02.786998 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.787028 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:02.787040 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.787146 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.787254 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.787301 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:03:02.788223 14379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:02.790527 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.790034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51528)
0928 15:03:02.790224 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.790337 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:03:02.791648 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.791177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51540)
0928 15:03:02.791356 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.791464 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:03:02.872620 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.871934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51542)
0928 15:03:02.872312 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.872450 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260928 15:03:02.873891 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.873387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36115 (local address 127.0.0.1:51556)
0928 15:03:02.873630 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.873729 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 15:03:02.874907 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14256
I20260928 15:03:02.890455 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 13902
I20260928 15:03:02.891615 13964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:02.997017 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 13902
2026-09-28T15:03:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2758 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 15:03:03.037521 9267 mini_oidc.cc:152] Starting JWKS server
I20260928 15:03:03.040158 9267 webserver.cc:460] Webserver started at https://127.0.0.1:45967/ using document root <none> and password file <none>
I20260928 15:03:03.040563 9267 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:03:03.041206 9267 webserver.cc:460] Webserver started at http://127.0.0.1:35835/ using document root <none> and password file <none>
2026-09-28T15:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:03Z Disabled control of system clock
I20260928 15:03:03.067030 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45763
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:45763
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45967/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/testchainca.cert with env {}
W20260928 15:03:03.378999 14440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:03.379341 14440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:03.379516 14440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:03.386636 14440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:03.386780 14440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:03.386873 14440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:03.387012 14440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:03.387094 14440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:03:03.387180 14440 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:03:03.387259 14440 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45967/jwks.json
I20260928 15:03:03.397070 14440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:45763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45967/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:03.398343 14440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:03.399637 14440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:03.411309 14445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.411305 14448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.411514 14446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.411957 14440 server_base.cc:1034] running on GCE node
I20260928 15:03:03.412779 14440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:03.414183 14440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:03.415405 14440 hybrid_clock.cc:648] HybridClock initialized: now 1790607783415372 us; error 51 us; skew 500 ppm
I20260928 15:03:03.415861 14440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:03.418105 14440 webserver.cc:460] Webserver started at http://127.9.12.254:42339/ using document root <none> and password file <none>
I20260928 15:03:03.418774 14440 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.418903 14440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.419226 14440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:03.422019 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "b2db7bc5650a4834a0d3fbcae01877c6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-dmhs"
server_key: "aa7c59a16f08646d76afbb14bdf73769"
server_key_iv: "815d467eec0f53d616337fb92e271906"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.422798 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "b2db7bc5650a4834a0d3fbcae01877c6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-dmhs"
server_key: "aa7c59a16f08646d76afbb14bdf73769"
server_key_iv: "815d467eec0f53d616337fb92e271906"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.427809 14440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 15:03:03.431300 14454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.432598 14440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:03:03.432783 14440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "b2db7bc5650a4834a0d3fbcae01877c6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-dmhs"
server_key: "aa7c59a16f08646d76afbb14bdf73769"
server_key_iv: "815d467eec0f53d616337fb92e271906"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.433022 14440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.447762 14440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:03.448643 14440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:03.448935 14440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:03.467010 14440 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:45763
I20260928 15:03:03.467015 14505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:45763 every 8 connection(s)
I20260928 15:03:03.468349 14440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:03.472013 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:03.474363 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14440
I20260928 15:03:03.474607 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:03.475427 9267 external_mini_cluster.cc:1428] Setting key 8056738b45224e475c85913e97dd1d43
I20260928 15:03:03.485853 14506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6: Bootstrap starting.
I20260928 15:03:03.489041 14506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:03.490077 14506 log.cc:824] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:03.491238 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.477194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.9.12.254:45763)
0928 15:03:03.478138 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.478150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.480836 (+ 2686us) server_negotiation.cc:407] Connection header received
0928 15:03:03.482554 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.482574 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.483031 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.483421 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.484021 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.485339 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.486675 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.487385 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.487597 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.488889 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.488896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.488909 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:03.489543 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.489629 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:03.489649 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.489777 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.489989 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.490326 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":376,"threads_started":1}
I20260928 15:03:03.493093 14506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6: No bootstrap required, opened a new log
I20260928 15:03:03.495221 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:03:03.501076 14506 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45763 } }
I20260928 15:03:03.501444 14506 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:03.501571 14506 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2db7bc5650a4834a0d3fbcae01877c6, State: Initialized, Role: FOLLOWER
I20260928 15:03:03.502169 14506 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45763 } }
I20260928 15:03:03.502363 14506 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:03.502460 14506 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:03.502576 14506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:03.505120 14506 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45763 } }
I20260928 15:03:03.505549 14506 leader_election.cc:302] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2db7bc5650a4834a0d3fbcae01877c6; no voters:
I20260928 15:03:03.506107 14506 leader_election.cc:288] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:03.506260 14511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:03.506974 14511 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [term 1 LEADER]: Becoming Leader. State: Replica: b2db7bc5650a4834a0d3fbcae01877c6, State: Running, Role: LEADER
I20260928 15:03:03.507521 14511 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45763 } }
I20260928 15:03:03.508324 14506 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:03.510229 14513 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2db7bc5650a4834a0d3fbcae01877c6. Latest consensus state: current_term: 1 leader_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45763 } } }
I20260928 15:03:03.510567 14513 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:03.511435 14517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:03.511296 14512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2db7bc5650a4834a0d3fbcae01877c6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45763 } } }
I20260928 15:03:03.511585 14512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:03.516497 14517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:03.525031 14517 catalog_manager.cc:1347] Generated new cluster ID: e518a2d05fec471bb37451a8d38d3c3d
I20260928 15:03:03.525146 14517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:03.535727 14517 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:03.537168 14517 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:03.547223 14517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2db7bc5650a4834a0d3fbcae01877c6: Generated new TSK 0
I20260928 15:03:03.548038 14517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:03:03.588421 14536 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 15:03:03.588598 14536 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:03:03.589475 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.562330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.9.12.254:45763)
0928 15:03:03.562535 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.562542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.562926 (+ 384us) server_negotiation.cc:407] Connection header received
0928 15:03:03.563269 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.563301 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.563366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.563537 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:03:03.564045 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.564832 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.566185 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.566615 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.566762 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.567284 (+ 522us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:03:03.588796 (+ 21512us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:03:03.589180 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 15:03:03.589608 14509 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 15:03:03.591111 14535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:03.562249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45763 (local address 127.0.0.1:35108)
0928 15:03:03.562830 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.562847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.563033 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.563578 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.563584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.563596 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:03:03.563896 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.563904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.564963 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.564970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.566051 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.566061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.566781 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.566787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.566942 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.567019 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:03:03.589141 (+ 22122us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:03:03.589488 (+ 347us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.12.254:45763: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":325,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.12.254:45763: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 15:03:03.593321 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14440
I20260928 15:03:03.594413 14501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:03.686610 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14440
2026-09-28T15:03:03Z chronyd exiting
I20260928 15:03:03.737191 9267 test_util.cc:175] -----------------------------------------------
I20260928 15:03:03.737336 9267 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607741729566-9267-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (704 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 15:03:03.741351 9267 mini_oidc.cc:152] Starting JWKS server
I20260928 15:03:03.743247 9267 webserver.cc:460] Webserver started at https://127.0.0.1:34041/ using document root <none> and password file <none>
I20260928 15:03:03.743461 9267 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:03:03.744109 9267 webserver.cc:460] Webserver started at http://127.0.0.1:40589/ using document root <none> and password file <none>
2026-09-28T15:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:03Z Disabled control of system clock
I20260928 15:03:03.771339 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:38827
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:38827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34041/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/testchainca.cert with env {}
W20260928 15:03:04.092811 14543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:04.093108 14543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:04.093304 14543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:04.100723 14543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:04.100824 14543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:04.100930 14543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:04.101038 14543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:04.101120 14543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:03:04.101263 14543 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:03:04.101321 14543 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34041/jwks.json
I20260928 15:03:04.111598 14543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:38827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34041/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:04.112921 14543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:04.114230 14543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:04.126034 14548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.126004 14551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.126108 14549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.126782 14543 server_base.cc:1034] running on GCE node
I20260928 15:03:04.127585 14543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:04.129061 14543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:04.130293 14543 hybrid_clock.cc:648] HybridClock initialized: now 1790607784130259 us; error 46 us; skew 500 ppm
I20260928 15:03:04.130652 14543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:04.132903 14543 webserver.cc:460] Webserver started at http://127.9.12.254:35631/ using document root <none> and password file <none>
I20260928 15:03:04.133622 14543 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.133762 14543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.134092 14543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:04.137009 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6"
format_stamp: "Formatted at 2026-09-28 15:03:04 on dist-test-slave-dmhs"
server_key: "8ee6642744dd97238ff16f74f5abf42e"
server_key_iv: "bf0db829c8d1cbf1f5072d6525db0c93"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.137807 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6"
format_stamp: "Formatted at 2026-09-28 15:03:04 on dist-test-slave-dmhs"
server_key: "8ee6642744dd97238ff16f74f5abf42e"
server_key_iv: "bf0db829c8d1cbf1f5072d6525db0c93"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.142947 14543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:03:04.146274 14557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.147643 14543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:04.147825 14543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6"
format_stamp: "Formatted at 2026-09-28 15:03:04 on dist-test-slave-dmhs"
server_key: "8ee6642744dd97238ff16f74f5abf42e"
server_key_iv: "bf0db829c8d1cbf1f5072d6525db0c93"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.148049 14543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.186121 14543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:04.187028 14543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:04.187353 14543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:04.206264 14543 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:38827
I20260928 15:03:04.206285 14609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:38827 every 8 connection(s)
I20260928 15:03:04.207675 14543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:04.210207 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14543
I20260928 15:03:04.210407 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:04.211055 9267 external_mini_cluster.cc:1428] Setting key a4cc4e0d6ef7bd09a5db455edf81de04
I20260928 15:03:04.211519 14610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:04.225791 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.213171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.9.12.254:38827)
0928 15:03:04.214406 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.214428 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.214592 (+ 164us) server_negotiation.cc:407] Connection header received
0928 15:03:04.216445 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.216475 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.217156 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.217623 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.218232 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.219562 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.221397 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.222046 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.222185 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.223504 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.223511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.223527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:04.224104 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.224194 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:04.224214 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.224360 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.224544 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.224873 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":424,"threads_started":1}
I20260928 15:03:04.226162 14610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6: Bootstrap starting.
I20260928 15:03:04.229779 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:03:04.229919 14610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:04.231108 14610 log.cc:824] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:04.234087 14610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6: No bootstrap required, opened a new log
I20260928 15:03:04.241173 14610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38827 } }
I20260928 15:03:04.241520 14610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:04.241596 14610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2db6d6bb3d408c8a2a4f37e3ca20e6, State: Initialized, Role: FOLLOWER
I20260928 15:03:04.242082 14610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38827 } }
I20260928 15:03:04.242277 14610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:04.242373 14610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:04.242501 14610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:04.244688 14610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38827 } }
I20260928 15:03:04.245153 14610 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2db6d6bb3d408c8a2a4f37e3ca20e6; no voters:
I20260928 15:03:04.245800 14610 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:04.245958 14615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:04.247786 14610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:04.249437 14615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2db6d6bb3d408c8a2a4f37e3ca20e6, State: Running, Role: LEADER
I20260928 15:03:04.250098 14615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38827 } }
I20260928 15:03:04.253270 14617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2db6d6bb3d408c8a2a4f37e3ca20e6. Latest consensus state: current_term: 1 leader_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38827 } } }
I20260928 15:03:04.253626 14617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.254225 14616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2db6d6bb3d408c8a2a4f37e3ca20e6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38827 } } }
I20260928 15:03:04.254499 14616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.254607 14627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:04.264895 14627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:04.273056 14627 catalog_manager.cc:1347] Generated new cluster ID: 96f4ce6fc6a9482da7cf586f9485c061
I20260928 15:03:04.273175 14627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:04.297434 14627 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:04.299271 14627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:04.312211 14627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2db6d6bb3d408c8a2a4f37e3ca20e6: Generated new TSK 0
I20260928 15:03:04.313306 14627 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:03:04.356681 14641 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 15:03:04.356812 14641 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:03:04.357892 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.337419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.9.12.254:38827)
0928 15:03:04.337641 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.337651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.338151 (+ 500us) server_negotiation.cc:407] Connection header received
0928 15:03:04.338408 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.338415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.338484 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.338609 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:03:04.339127 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.339859 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.341920 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.342547 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.342642 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.343224 (+ 582us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:03:04.357359 (+ 14135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:03:04.357659 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 15:03:04.358043 14613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 15:03:04.358295 14640 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:04.337370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:38827 (local address 127.0.0.1:58288)
0928 15:03:04.338044 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.338065 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.338215 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.338649 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.338655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.338667 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:03:04.338954 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.338974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.340015 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.340022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.341751 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.341766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.342730 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.342737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.342927 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.342983 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:03:04.357601 (+ 14618us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:03:04.357877 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.12.254:38827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":348,"threads_started":1}
I20260928 15:03:04.359334 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14543
I20260928 15:03:04.360543 14605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:04.459933 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14543
2026-09-28T15:03:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (773 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 15:03:04.515134 9267 mini_oidc.cc:152] Starting JWKS server
I20260928 15:03:04.517066 9267 webserver.cc:460] Webserver started at https://127.0.0.1:36415/ using document root <none> and password file <none>
I20260928 15:03:04.517316 9267 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:03:04.517931 9267 webserver.cc:460] Webserver started at http://127.0.0.1:33635/ using document root <none> and password file <none>
2026-09-28T15:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:04Z Disabled control of system clock
I20260928 15:03:04.545006 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:36639
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:36639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36415/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:03:04.864686 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:04.865042 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:04.865206 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:04.872490 14648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:04.872629 14648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:04.872720 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:04.872830 14648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:04.872905 14648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:03:04.873018 14648 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:03:04.873096 14648 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36415/jwks.json
I20260928 15:03:04.882081 14648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:36639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36415/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:04.883360 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:04.885236 14648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:04.896670 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.896713 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.896746 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.897571 14648 server_base.cc:1034] running on GCE node
I20260928 15:03:04.898332 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:04.899665 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:04.900873 14648 hybrid_clock.cc:648] HybridClock initialized: now 1790607784900806 us; error 55 us; skew 500 ppm
I20260928 15:03:04.901204 14648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:04.903287 14648 webserver.cc:460] Webserver started at http://127.9.12.254:38331/ using document root <none> and password file <none>
I20260928 15:03:04.903899 14648 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.903991 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.904245 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:04.907107 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "c3767fadda624752a4ae334ebfc692de"
format_stamp: "Formatted at 2026-09-28 15:03:04 on dist-test-slave-dmhs"
server_key: "f865115382dacf7c87639e5b3154ce46"
server_key_iv: "2534b6eb6183cb4496187d3ffbeb7d00"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.907828 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "c3767fadda624752a4ae334ebfc692de"
format_stamp: "Formatted at 2026-09-28 15:03:04 on dist-test-slave-dmhs"
server_key: "f865115382dacf7c87639e5b3154ce46"
server_key_iv: "2534b6eb6183cb4496187d3ffbeb7d00"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.912843 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:03:04.916226 14662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.917661 14648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:04.917868 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "c3767fadda624752a4ae334ebfc692de"
format_stamp: "Formatted at 2026-09-28 15:03:04 on dist-test-slave-dmhs"
server_key: "f865115382dacf7c87639e5b3154ce46"
server_key_iv: "2534b6eb6183cb4496187d3ffbeb7d00"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.918151 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.935619 14648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:04.936458 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:04.936764 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:04.955931 14648 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:36639
I20260928 15:03:04.955951 14714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:36639 every 8 connection(s)
I20260928 15:03:04.957307 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:04.960999 14715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:04.963127 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14648
I20260928 15:03:04.963347 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:04.964140 9267 external_mini_cluster.cc:1428] Setting key d24f3b79a8f0e556ad49b4711b7ee46c
I20260928 15:03:04.972651 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de: Bootstrap starting.
I20260928 15:03:04.975756 14715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:04.976845 14715 log.cc:824] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:04.977092 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.966109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.9.12.254:36639)
0928 15:03:04.967087 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.967098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.967137 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:03:04.968828 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.968850 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.969276 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.969655 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.970197 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.971443 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.972769 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.973429 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.973648 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.974802 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.974809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.974822 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:04.975323 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.975407 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:04.975426 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.975565 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.975799 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.976124 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":334,"threads_started":1}
I20260928 15:03:04.980154 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de: No bootstrap required, opened a new log
I20260928 15:03:04.981199 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:03:04.988817 14715 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3767fadda624752a4ae334ebfc692de" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36639 } }
I20260928 15:03:04.989151 14715 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:04.989226 14715 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3767fadda624752a4ae334ebfc692de, State: Initialized, Role: FOLLOWER
I20260928 15:03:04.989704 14715 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3767fadda624752a4ae334ebfc692de" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36639 } }
I20260928 15:03:04.989949 14715 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:04.990115 14715 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:04.990304 14715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:04.992486 14715 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3767fadda624752a4ae334ebfc692de" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36639 } }
I20260928 15:03:04.992987 14715 leader_election.cc:302] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3767fadda624752a4ae334ebfc692de; no voters:
I20260928 15:03:04.993539 14715 leader_election.cc:288] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:04.993736 14720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:04.994438 14720 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [term 1 LEADER]: Becoming Leader. State: Replica: c3767fadda624752a4ae334ebfc692de, State: Running, Role: LEADER
I20260928 15:03:04.994941 14720 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3767fadda624752a4ae334ebfc692de" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36639 } }
I20260928 15:03:04.995870 14715 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:04.998026 14722 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3767fadda624752a4ae334ebfc692de. Latest consensus state: current_term: 1 leader_uuid: "c3767fadda624752a4ae334ebfc692de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3767fadda624752a4ae334ebfc692de" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36639 } } }
I20260928 15:03:04.998498 14722 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.998230 14721 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3767fadda624752a4ae334ebfc692de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3767fadda624752a4ae334ebfc692de" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36639 } } }
I20260928 15:03:04.998874 14721 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:05.000576 14725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:05.007143 14725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:05.018042 14725 catalog_manager.cc:1347] Generated new cluster ID: a2e5365c465846f9b496c3027c0fcddf
I20260928 15:03:05.018165 14725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:05.030833 14725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:05.032573 14725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:05.041364 14725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3767fadda624752a4ae334ebfc692de: Generated new TSK 0
I20260928 15:03:05.042359 14725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:05.067651 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:05.059090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.9.12.254:36639)
0928 15:03:05.059319 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:05.059325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:05.059689 (+ 364us) server_negotiation.cc:407] Connection header received
0928 15:03:05.059865 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:05.059870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:05.059964 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:05.060123 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:03:05.060596 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.061684 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.063009 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.063851 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.064033 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:05.064586 (+ 553us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:03:05.066991 (+ 2405us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:03:05.067321 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39536: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 15:03:05.067776 14718 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39536: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 15:03:05.069298 14744 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:05.058953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36639 (local address 127.0.0.1:39536)
0928 15:03:05.059593 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:05.059608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:05.059717 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:05.060138 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:05.060141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:05.060150 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:03:05.060450 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:05.060460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.061829 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.061832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:05.062849 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:05.062859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.064099 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.064103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:05.064240 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:05.064327 (+ 87us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:03:05.067209 (+ 2882us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:03:05.068876 (+ 1667us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.12.254:36639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":1379,"thread_start_us":367,"threads_started":1}
I20260928 15:03:05.071161 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14648
I20260928 15:03:05.072283 14710 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 15:03:05.079218 14745 webserver.cc:533] Webserver: sslize failed:
E20260928 15:03:05.079345 14745 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:03:05.159698 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14648
2026-09-28T15:03:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T15:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:05Z Disabled control of system clock
I20260928 15:03:05.237552 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34103
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:34103
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/testchainca.cert with env {}
W20260928 15:03:05.560097 14750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:05.560452 14750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:05.560577 14750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:05.567816 14750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:05.567924 14750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:05.567961 14750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:05.568011 14750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:05.568060 14750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:03:05.568120 14750 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:03:05.568161 14750 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 15:03:05.578395 14750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:34103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:05.579681 14750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:05.580971 14750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:05.592665 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:05.592940 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:05.593297 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:05.593782 14750 server_base.cc:1034] running on GCE node
I20260928 15:03:05.594537 14750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:05.595916 14750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:05.597162 14750 hybrid_clock.cc:648] HybridClock initialized: now 1790607785597115 us; error 59 us; skew 500 ppm
I20260928 15:03:05.597529 14750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:05.599699 14750 webserver.cc:460] Webserver started at http://127.9.12.254:45127/ using document root <none> and password file <none>
I20260928 15:03:05.600543 14750 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:05.600678 14750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:05.600981 14750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:05.603794 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "b11763e714db4f3cb6d9c583e09963cf"
format_stamp: "Formatted at 2026-09-28 15:03:05 on dist-test-slave-dmhs"
server_key: "a36bf7b8c1094406804543ed282926c2"
server_key_iv: "fda392da523bbf4f6007447b42dbd957"
server_key_version: "encryptionkey@0"
I20260928 15:03:05.604682 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "b11763e714db4f3cb6d9c583e09963cf"
format_stamp: "Formatted at 2026-09-28 15:03:05 on dist-test-slave-dmhs"
server_key: "a36bf7b8c1094406804543ed282926c2"
server_key_iv: "fda392da523bbf4f6007447b42dbd957"
server_key_version: "encryptionkey@0"
I20260928 15:03:05.609606 14750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:03:05.612987 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:05.614298 14750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:05.614444 14750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "b11763e714db4f3cb6d9c583e09963cf"
format_stamp: "Formatted at 2026-09-28 15:03:05 on dist-test-slave-dmhs"
server_key: "a36bf7b8c1094406804543ed282926c2"
server_key_iv: "fda392da523bbf4f6007447b42dbd957"
server_key_version: "encryptionkey@0"
I20260928 15:03:05.614632 14750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:05.628796 14750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:05.629556 14750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:05.629786 14750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:05.647683 14750 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:34103
I20260928 15:03:05.647701 14815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:34103 every 8 connection(s)
I20260928 15:03:05.649152 14750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:05.653158 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:05.656194 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14750
I20260928 15:03:05.656433 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:05.657291 9267 external_mini_cluster.cc:1428] Setting key 8941dd92eb236e2caa6f69c702030ce8
I20260928 15:03:05.665333 14816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf: Bootstrap starting.
I20260928 15:03:05.668509 14816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:05.670015 14816 log.cc:824] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:05.671590 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:05.659295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.9.12.254:34103)
0928 15:03:05.660234 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:05.660252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:05.660351 (+ 99us) server_negotiation.cc:407] Connection header received
0928 15:03:05.661992 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:05.662016 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:05.662502 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:05.662901 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:05.663486 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.664801 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.666126 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.666813 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.667037 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:05.668308 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:05.668320 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:05.668340 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:05.669091 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:05.669219 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:05.669246 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:05.669357 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:05.670060 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 15:03:05.670355 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":306,"threads_started":1}
I20260928 15:03:05.673223 14816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf: No bootstrap required, opened a new log
I20260928 15:03:05.675380 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:03:05.681073 14816 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11763e714db4f3cb6d9c583e09963cf" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34103 } }
I20260928 15:03:05.681555 14816 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:05.681679 14816 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11763e714db4f3cb6d9c583e09963cf, State: Initialized, Role: FOLLOWER
I20260928 15:03:05.682312 14816 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11763e714db4f3cb6d9c583e09963cf" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34103 } }
I20260928 15:03:05.682543 14816 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:05.682691 14816 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:05.682862 14816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:05.685070 14816 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11763e714db4f3cb6d9c583e09963cf" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34103 } }
I20260928 15:03:05.685573 14816 leader_election.cc:302] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11763e714db4f3cb6d9c583e09963cf; no voters:
I20260928 15:03:05.686167 14816 leader_election.cc:288] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:05.686312 14821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:05.687108 14821 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [term 1 LEADER]: Becoming Leader. State: Replica: b11763e714db4f3cb6d9c583e09963cf, State: Running, Role: LEADER
I20260928 15:03:05.687589 14821 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11763e714db4f3cb6d9c583e09963cf" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34103 } }
I20260928 15:03:05.688474 14816 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:05.690359 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11763e714db4f3cb6d9c583e09963cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11763e714db4f3cb6d9c583e09963cf" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34103 } } }
I20260928 15:03:05.690888 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:05.691717 14829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:05.693199 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11763e714db4f3cb6d9c583e09963cf. Latest consensus state: current_term: 1 leader_uuid: "b11763e714db4f3cb6d9c583e09963cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11763e714db4f3cb6d9c583e09963cf" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 34103 } } }
I20260928 15:03:05.693694 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:05.699612 14829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:05.707626 14829 catalog_manager.cc:1347] Generated new cluster ID: aa6ef5217fa844828fb3e1f48c2412f2
I20260928 15:03:05.707736 14829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:05.733042 14829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:05.734972 14829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:05.757227 14829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11763e714db4f3cb6d9c583e09963cf: Generated new TSK 0
I20260928 15:03:05.758111 14829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:06.255239 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:05.773071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.9.12.254:34103)
0928 15:03:05.773315 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:05.773325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:05.773764 (+ 439us) server_negotiation.cc:407] Connection header received
0928 15:03:05.773990 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:05.773995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:05.774077 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:05.774208 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:03:05.774778 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.775492 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.777607 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.778079 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.778175 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:05.778756 (+ 581us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:03:06.254680 (+475924us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:03:06.255017 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36404: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 15:03:06.255384 14819 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36404: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 15:03:06.255692 14845 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:05.773011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:34103 (local address 127.0.0.1:36404)
0928 15:03:05.773650 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:05.773670 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:05.773825 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:05.774257 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:05.774262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:05.774276 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:03:05.774624 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:05.774638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.775647 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.775652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:05.777458 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:05.777475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:05.778257 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:05.778264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:05.778466 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:05.778516 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:03:06.254998 (+476482us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:03:06.255252 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.12.254:34103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":322,"threads_started":1}
I20260928 15:03:06.256985 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14750
I20260928 15:03:06.258037 14811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:06.359750 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14750
2026-09-28T15:03:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1184 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (394 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:07 dist-test-slave-dmhs krb5kdc[14860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:07 dist-test-slave-dmhs krb5kdc[14860](info): set up 2 sockets
Sep 28 15:03:07 dist-test-slave-dmhs krb5kdc[14860](info): commencing operation
krb5kdc: starting...
W20260928 15:03:08.233186 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:03:08 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:03:08.445959 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:44243
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:44243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:08.769148 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:08.769423 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:08.769598 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:08.776530 14873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:08.776667 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:08.776784 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:08.776881 14873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:08.776968 14873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:08.786083 14873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:44243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:08.787431 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:08.788754 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:08.801186 14881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:08.801143 14879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:08.801288 14878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:08.802157 14873 server_base.cc:1034] running on GCE node
I20260928 15:03:08.802961 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:08.804373 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:08.805604 14873 hybrid_clock.cc:648] HybridClock initialized: now 1790607788805557 us; error 52 us; skew 500 ppm
I20260928 15:03:08.806022 14873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:08 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607788, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:08.809048 14873 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:03:08.810477 14873 webserver.cc:460] Webserver started at http://127.9.12.254:33947/ using document root <none> and password file <none>
I20260928 15:03:08.811113 14873 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:08.811244 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:08.811538 14873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:08.814409 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "3c110b653dd240468c2aba2f8169fefb"
format_stamp: "Formatted at 2026-09-28 15:03:08 on dist-test-slave-dmhs"
server_key: "a02e8922050125ed38aa69b3b72e8e2a"
server_key_iv: "af4d91e65108f6c2281c8ffbdf954d47"
server_key_version: "encryptionkey@0"
I20260928 15:03:08.815165 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "3c110b653dd240468c2aba2f8169fefb"
format_stamp: "Formatted at 2026-09-28 15:03:08 on dist-test-slave-dmhs"
server_key: "a02e8922050125ed38aa69b3b72e8e2a"
server_key_iv: "af4d91e65108f6c2281c8ffbdf954d47"
server_key_version: "encryptionkey@0"
I20260928 15:03:08.819939 14873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 15:03:08.823148 14888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:08.824431 14873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:08.824600 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "3c110b653dd240468c2aba2f8169fefb"
format_stamp: "Formatted at 2026-09-28 15:03:08 on dist-test-slave-dmhs"
server_key: "a02e8922050125ed38aa69b3b72e8e2a"
server_key_iv: "af4d91e65108f6c2281c8ffbdf954d47"
server_key_version: "encryptionkey@0"
I20260928 15:03:08.824790 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:08.841234 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:08.844815 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:08.845101 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:08.863920 14873 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:44243
I20260928 15:03:08.863955 14939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:44243 every 8 connection(s)
I20260928 15:03:08.865356 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:08.868520 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14873
I20260928 15:03:08.868757 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:08.869491 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:08.869619 9267 external_mini_cluster.cc:1428] Setting key 8a04a3082f2b0fc7128043999d04a400
I20260928 15:03:08.883577 14940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb: Bootstrap starting.
Sep 28 15:03:08 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:08.887387 14940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:08.888708 14940 log.cc:824] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:08.891842 14940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb: No bootstrap required, opened a new log
I20260928 15:03:08.892944 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.871572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.9.12.254:44243)
0928 15:03:08.872353 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:08.872364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:08.872405 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:03:08.874095 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:08.874115 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:08.874492 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:08.874788 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:08.876419 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:08.877642 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:08.879432 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:08.880116 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:08.880244 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:08.884870 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:08.884880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:08.884912 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:08.884961 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:08.888683 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:08.889436 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:08.889441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:08.889447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:08.889602 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:08.889989 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:08.889993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:08.889996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:08.890594 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:08.890878 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:08.891214 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:03:08.891626 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":347,"threads_started":1}
I20260928 15:03:08.899663 14940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c110b653dd240468c2aba2f8169fefb" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 44243 } }
I20260928 15:03:08.899957 14940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:08.900017 14940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c110b653dd240468c2aba2f8169fefb, State: Initialized, Role: FOLLOWER
I20260928 15:03:08.900517 14940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c110b653dd240468c2aba2f8169fefb" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 44243 } }
I20260928 15:03:08.900707 14940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:08.900796 14940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:08.900897 14940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:08.903017 14940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c110b653dd240468c2aba2f8169fefb" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 44243 } }
I20260928 15:03:08.903475 14940 leader_election.cc:302] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c110b653dd240468c2aba2f8169fefb; no voters:
I20260928 15:03:08.903965 14940 leader_election.cc:288] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:08.904103 14945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:08.904811 14945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [term 1 LEADER]: Becoming Leader. State: Replica: 3c110b653dd240468c2aba2f8169fefb, State: Running, Role: LEADER
I20260928 15:03:08.905352 14945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c110b653dd240468c2aba2f8169fefb" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 44243 } }
I20260928 15:03:08.906183 14940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:08.908564 14946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c110b653dd240468c2aba2f8169fefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c110b653dd240468c2aba2f8169fefb" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 44243 } } }
I20260928 15:03:08.908551 14947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c110b653dd240468c2aba2f8169fefb. Latest consensus state: current_term: 1 leader_uuid: "3c110b653dd240468c2aba2f8169fefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c110b653dd240468c2aba2f8169fefb" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 44243 } } }
I20260928 15:03:08.908960 14946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:08.908981 14947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:08.910066 14950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:08.914528 14950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:08.923313 14950 catalog_manager.cc:1347] Generated new cluster ID: 1c1aed6fe96c43378d000bc597667319
I20260928 15:03:08.923421 14950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:08.933305 14950 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:08.934652 14950 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:08.942293 14950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c110b653dd240468c2aba2f8169fefb: Generated new TSK 0
I20260928 15:03:08.942991 14950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:03:08.985816 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:44243
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:09.296628 14966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:09.296895 14966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:09.297050 14966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:09.304021 14966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:09.304119 14966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:09.304375 14966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:09.315276 14966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:44243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:09.316545 14966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:09.317806 14966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:09.330201 14974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.330288 14971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.330259 14972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:09.331161 14966 server_base.cc:1034] running on GCE node
I20260928 15:03:09.331740 14966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:09.332662 14966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:09.333873 14966 hybrid_clock.cc:648] HybridClock initialized: now 1790607789333830 us; error 28 us; skew 500 ppm
I20260928 15:03:09.334185 14966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:09 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:09.337144 14966 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:03:09.338456 14966 webserver.cc:460] Webserver started at http://127.9.12.193:35155/ using document root <none> and password file <none>
I20260928 15:03:09.339083 14966 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:09.339212 14966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:09.339546 14966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:09.342362 14966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "12e5c6815b2846a99457321a20482b4b"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-dmhs"
server_key: "62c5ca0b7420d504a29aac0d19b20763"
server_key_iv: "2c30d19ad9a015f411eb2e48138913a8"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.343273 14966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "12e5c6815b2846a99457321a20482b4b"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-dmhs"
server_key: "62c5ca0b7420d504a29aac0d19b20763"
server_key_iv: "2c30d19ad9a015f411eb2e48138913a8"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.348256 14966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:03:09.351296 14981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.352470 14966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260928 15:03:09.352626 14966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "12e5c6815b2846a99457321a20482b4b"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-dmhs"
server_key: "62c5ca0b7420d504a29aac0d19b20763"
server_key_iv: "2c30d19ad9a015f411eb2e48138913a8"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.352810 14966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:09.368409 14966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:09.371907 14966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:09.372169 14966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:09.373055 14966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:09.374346 14966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:09.374449 14966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.374610 14966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:09.374684 14966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.406781 14966 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:39147
I20260928 15:03:09.406801 15093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:39147 every 8 connection(s)
I20260928 15:03:09.408166 14966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:09.415028 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 14966
I20260928 15:03:09.415253 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:09.415818 9267 external_mini_cluster.cc:1428] Setting key 48efe0215e0aff2e88b0862733982d49
Sep 28 15:03:09 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:09.428618 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:09.410698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:55509 (local address 127.9.12.254:44243)
0928 15:03:09.411170 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:09.411176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:09.411898 (+ 722us) server_negotiation.cc:407] Connection header received
0928 15:03:09.413690 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:09.413694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:09.413773 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:09.413933 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:09.415418 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.416171 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.418164 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.418592 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.418726 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:09.422234 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:09.422243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:09.422251 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:09.422296 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:09.425348 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:09.426285 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:09.426289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:09.426291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:09.426405 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:09.426873 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:09.426876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:09.426878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:09.427142 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:09.427248 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:09.428123 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 15:03:09.428350 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":236,"threads_started":1}
I20260928 15:03:09.430075 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:09.411053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:44243 (local address 127.9.12.193:55509)
0928 15:03:09.411756 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:09.411795 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:09.413421 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:09.414163 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:09.414171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:09.414650 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:09.415255 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:09.415272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.416357 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.416361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:09.418010 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:09.418021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.418818 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.418827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:09.419054 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:09.419711 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:09.419734 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:09.421992 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:09.425489 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:09.425501 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:09.425520 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:09.426167 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:09.426506 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:09.426514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:09.426521 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:09.426763 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:09.427273 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:09.427284 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:09.427712 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 15:03:09.427999 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 15:03:09.428618 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":261,"threads_started":1}
I20260928 15:03:09.431679 15094 heartbeater.cc:347] Connected to a master server at 127.9.12.254:44243
I20260928 15:03:09.431995 15094 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:09.432896 15094 heartbeater.cc:511] Master 127.9.12.254:44243 requested a full tablet report, sending...
I20260928 15:03:09.435437 14905 ts_manager.cc:194] Registered new tserver with Master: 12e5c6815b2846a99457321a20482b4b (127.9.12.193:39147)
I20260928 15:03:09.437623 14905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:55509
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:03:09.463991 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:44243
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:09.774665 15100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:09.774997 15100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:09.775169 15100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:09.782166 15100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:09.782316 15100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:09.782482 15100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:09.792372 15100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:44243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:09.793700 15100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:09.795357 15100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:09.808475 15105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.808451 15106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.808943 15108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:09.809161 15100 server_base.cc:1034] running on GCE node
I20260928 15:03:09.809836 15100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:09.810837 15100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:09.812060 15100 hybrid_clock.cc:648] HybridClock initialized: now 1790607789812026 us; error 76 us; skew 500 ppm
I20260928 15:03:09.812412 15100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:09 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:09.815366 15100 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:03:09.816792 15100 webserver.cc:460] Webserver started at http://127.9.12.194:44565/ using document root <none> and password file <none>
I20260928 15:03:09.817448 15100 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:09.817595 15100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:09.817891 15100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:09.820744 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "46d0ba3b518c40a2a440854f8fe681f9"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-dmhs"
server_key: "51bc69a7dbcbd47dd695d1447a92a994"
server_key_iv: "d02fac504480cb3a6ec46cee4b834cc5"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.821462 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "46d0ba3b518c40a2a440854f8fe681f9"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-dmhs"
server_key: "51bc69a7dbcbd47dd695d1447a92a994"
server_key_iv: "d02fac504480cb3a6ec46cee4b834cc5"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.826661 15100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:03:09.831284 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.832881 15100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:09.833029 15100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "46d0ba3b518c40a2a440854f8fe681f9"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-dmhs"
server_key: "51bc69a7dbcbd47dd695d1447a92a994"
server_key_iv: "d02fac504480cb3a6ec46cee4b834cc5"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.833220 15100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:09.850299 15100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:09.854661 15100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:09.854957 15100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:09.855944 15100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:09.857540 15100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:09.857637 15100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.857788 15100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:09.857857 15100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.889894 15100 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:45955
I20260928 15:03:09.889912 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:45955 every 8 connection(s)
I20260928 15:03:09.891484 15100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:09.892156 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15100
I20260928 15:03:09.892349 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:09.892939 9267 external_mini_cluster.cc:1428] Setting key 7b96438df1e1fe57fcbffb6e50b883be
Sep 28 15:03:09 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:09.913552 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:09.894378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:39889 (local address 127.9.12.254:44243)
0928 15:03:09.894545 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:09.894550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:09.896028 (+ 1478us) server_negotiation.cc:407] Connection header received
0928 15:03:09.897863 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:09.897871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:09.897959 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:09.898078 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:09.899466 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.900651 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.903097 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.903500 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.903603 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:09.907684 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:09.907689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:09.907692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:09.907722 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:09.910464 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:09.911238 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:09.911251 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:09.911254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:09.911392 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:09.911817 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:09.911822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:09.911825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:09.912090 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:09.912256 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:09.912963 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 15:03:09.913192 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:03:09.914577 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:09.895031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:44243 (local address 127.9.12.194:39889)
0928 15:03:09.895858 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:09.895905 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:09.897607 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:09.898273 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:09.898280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:09.898754 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:09.899300 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:09.899316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.900799 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.900803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:09.902942 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:09.902959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.903720 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.903729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:09.904083 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:09.904797 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:09.904829 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:09.907481 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:09.910606 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:09.910615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:09.910632 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:09.911112 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:09.911506 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:09.911512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:09.911516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:09.911703 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:09.912228 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:09.912235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:09.912611 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 15:03:09.912876 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 15:03:09.913406 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":353,"threads_started":1}
I20260928 15:03:09.916079 15228 heartbeater.cc:347] Connected to a master server at 127.9.12.254:44243
I20260928 15:03:09.916420 15228 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
I20260928 15:03:09.917227 15228 heartbeater.cc:511] Master 127.9.12.254:44243 requested a full tablet report, sending...
I20260928 15:03:09.918697 14905 ts_manager.cc:194] Registered new tserver with Master: 46d0ba3b518c40a2a440854f8fe681f9 (127.9.12.194:45955)
I20260928 15:03:09.919483 14905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:39889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:03:09.939749 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:44243
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:10.254339 15233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:10.254613 15233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:10.254709 15233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:10.262037 15233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:10.262140 15233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:10.262245 15233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:10.271504 15233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:44243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:10.272689 15233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:10.273875 15233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:10.286937 15239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:10.287154 15238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:10.287173 15241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:10.288012 15233 server_base.cc:1034] running on GCE node
I20260928 15:03:10.288657 15233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:10.289567 15233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:10.290797 15233 hybrid_clock.cc:648] HybridClock initialized: now 1790607790290759 us; error 50 us; skew 500 ppm
I20260928 15:03:10.291102 15233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.294219 15233 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:03:10.295609 15233 webserver.cc:460] Webserver started at http://127.9.12.195:34347/ using document root <none> and password file <none>
I20260928 15:03:10.296336 15233 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:10.296485 15233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:10.296787 15233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:10.299477 15233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "d0b6396a89104b5ebf574288c8d71526"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-dmhs"
server_key: "5cc28029f1497b4cc4943edab4141431"
server_key_iv: "c1e864f27d4592bc4660845787295410"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.300217 15233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "d0b6396a89104b5ebf574288c8d71526"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-dmhs"
server_key: "5cc28029f1497b4cc4943edab4141431"
server_key_iv: "c1e864f27d4592bc4660845787295410"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.305294 15233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:03:10.308841 15248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.310169 15233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:10.310336 15233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "d0b6396a89104b5ebf574288c8d71526"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-dmhs"
server_key: "5cc28029f1497b4cc4943edab4141431"
server_key_iv: "c1e864f27d4592bc4660845787295410"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.310540 15233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:10.339720 15233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:10.344030 15233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:10.344409 15233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:10.345391 15233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:10.346825 15233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:10.346946 15233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.347101 15233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:10.347188 15233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.379985 15233 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:39059
I20260928 15:03:10.380029 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:39059 every 8 connection(s)
I20260928 15:03:10.381282 15233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:10.389094 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15233
I20260928 15:03:10.389264 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:10.389796 9267 external_mini_cluster.cc:1428] Setting key 76e8aa03db635166eebe14f09e3e3e1b
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:10.400853 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.383685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:44507 (local address 127.9.12.254:44243)
0928 15:03:10.383890 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.383914 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.385238 (+ 1324us) server_negotiation.cc:407] Connection header received
0928 15:03:10.387003 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.387007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.387075 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.387162 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.388672 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.389454 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.391566 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.392171 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.392343 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.395751 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.395756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.395759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.395789 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.398362 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.399011 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.399014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.399016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.399125 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.399468 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.399471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.399473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.399711 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.399820 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.400469 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.400617 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:03:10.401790 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.384324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:44243 (local address 127.9.12.195:44507)
0928 15:03:10.385063 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.385106 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.386722 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.387380 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.387387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.387887 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.388522 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.388541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.389622 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.389627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.391428 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.391442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.392443 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.392448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.392714 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.393447 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.393469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.395573 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.398473 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.398479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.398490 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.398906 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.399213 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.399216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.399218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.399384 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.399813 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.399819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.400116 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.400387 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.400752 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":321,"threads_started":1}
I20260928 15:03:10.403018 15361 heartbeater.cc:347] Connected to a master server at 127.9.12.254:44243
I20260928 15:03:10.403359 15361 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:10.404165 15361 heartbeater.cc:511] Master 127.9.12.254:44243 requested a full tablet report, sending...
I20260928 15:03:10.405579 14905 ts_manager.cc:194] Registered new tserver with Master: d0b6396a89104b5ebf574288c8d71526 (127.9.12.195:39059)
I20260928 15:03:10.406661 14905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:44507
I20260928 15:03:10.417061 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:10.440822 15094 heartbeater.cc:503] Master 127.9.12.254:44243 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/test-admin.keytab.
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.444932 9267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 15:03:10.454311 15368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:10.449970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.9.12.193:39147)
0928 15:03:10.452741 (+ 2771us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.452748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.452773 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:03:10.452827 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.452833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.453036 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.453165 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.453943 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43058: BlockingRecv error: recv got EOF from 127.0.0.1:43058 (error 108)
Metrics: {"server-negotiator.queue_time_us":2630,"thread_start_us":2492,"threads_started":1}
W20260928 15:03:10.454473 9267 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.456187 9267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.457645 9267 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:03:10.472088 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.459104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.9.12.193:39147)
0928 15:03:10.459298 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.459305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.459542 (+ 237us) server_negotiation.cc:407] Connection header received
0928 15:03:10.459738 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.459744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.459807 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.459903 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.460962 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.461887 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.463086 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.463664 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.463796 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.467314 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.467324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.467338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.467369 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.469903 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.470473 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.470479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.470481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.470587 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.470863 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.470866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.470867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.471204 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.471405 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.471660 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.471839 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:03:10.477763 15368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:10.476034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.9.12.193:39147)
0928 15:03:10.476229 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.476234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.476475 (+ 241us) server_negotiation.cc:407] Connection header received
0928 15:03:10.476664 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.476669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.476742 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.476834 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.477540 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43078: BlockingRecv error: recv got EOF from 127.0.0.1:43078 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 15:03:10.478116 9267 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.479472 9267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.480785 9267 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:03:10.493765 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.481992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.9.12.193:39147)
0928 15:03:10.482199 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.482205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.482370 (+ 165us) server_negotiation.cc:407] Connection header received
0928 15:03:10.482478 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.482481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.482532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.482606 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.483474 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.484146 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.485315 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.485750 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.485836 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.489043 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.489048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.489051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.489088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.491650 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.492132 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.492136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.492138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.492363 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.492679 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.492682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.492684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.492917 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.493110 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.493309 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.493479 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 15:03:10.498035 15368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:10.496251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.9.12.193:39147)
0928 15:03:10.496477 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.496482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.496720 (+ 238us) server_negotiation.cc:407] Connection header received
0928 15:03:10.496899 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.496904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.496976 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.497076 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.497780 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43098: BlockingRecv error: recv got EOF from 127.0.0.1:43098 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 15:03:10.498188 9267 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607741729566-9267-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.499643 9267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.500880 9267 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:03:10.514259 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.502017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.9.12.193:39147)
0928 15:03:10.502174 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.502177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.502437 (+ 260us) server_negotiation.cc:407] Connection header received
0928 15:03:10.502613 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.502619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.502686 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.502771 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.503786 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.504538 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.505739 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.506197 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.506352 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.509454 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.509458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.509461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.509487 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.512002 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.512617 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.512621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.512623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.512772 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.513058 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.513069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.513071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.513345 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.513557 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.513770 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.513982 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:03:10.515827 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14966
I20260928 15:03:10.526432 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.516492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.9.12.193:39147)
0928 15:03:10.516655 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.516660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.517037 (+ 377us) server_negotiation.cc:407] Connection header received
0928 15:03:10.517159 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.517162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.517211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.517281 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.518145 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518769 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.519840 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.520233 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.520433 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.522171 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.522176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.522179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.522204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.524434 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.524966 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.524970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.524972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.525085 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.525405 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.525408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.525410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.525643 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.525818 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.526052 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.526194 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:03:10.527033 15089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:10.607975 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14966
I20260928 15:03:10.624934 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15100
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:03:10.645012 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.625898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.9.12.194:45955)
0928 15:03:10.626463 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.626469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.626494 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:03:10.626554 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.626562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.627032 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.627178 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.628189 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.629350 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.630515 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.631021 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.631123 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.639408 (+ 8285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.639419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.639433 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.639467 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.642171 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.642804 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.642819 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.642821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.642945 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.643305 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.643311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.643314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.643680 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.643995 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.644360 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.644612 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":326,"threads_started":1}
I20260928 15:03:10.646394 15223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:10.743213 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15100
I20260928 15:03:10.758690 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15233
Sep 28 15:03:10 dist-test-slave-dmhs krb5kdc[14860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:03:10.773815 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.759735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.9.12.195:39059)
0928 15:03:10.760434 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.760440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.760470 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:03:10.760535 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.760542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.760751 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.760878 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.761774 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.762770 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.763970 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.764582 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.764679 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.768553 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.768565 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.768579 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.768613 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.771358 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.771948 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.771954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.771956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.772067 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.772512 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.772515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.772517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.772877 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.773103 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.773332 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.773536 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":386,"threads_started":1}
I20260928 15:03:10.775116 15356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:10.857386 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15233
I20260928 15:03:10.875344 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 14873
I20260928 15:03:10.876565 14935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:10.976729 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 14873
2026-09-28T15:03:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3826 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:11 dist-test-slave-dmhs krb5kdc[15388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:11 dist-test-slave-dmhs krb5kdc[15388](info): set up 2 sockets
Sep 28 15:03:11 dist-test-slave-dmhs krb5kdc[15388](info): commencing operation
krb5kdc: starting...
W20260928 15:03:12.041123 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:03:12 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab.
Entry for principal oryx/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab.
I20260928 15:03:12.239667 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40395
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab
--principal=oryx/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:12.552677 15401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:12.553017 15401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:12.553184 15401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:12.559854 15401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:12.560001 15401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:12.560107 15401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:12.560197 15401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:12.560330 15401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:12.569830 15401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab
--principal=oryx/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:12.571094 15401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:12.572381 15401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:12.583918 15407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:12.583916 15406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:12.584198 15409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:12.584508 15401 server_base.cc:1034] running on GCE node
I20260928 15:03:12.585461 15401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:12.586736 15401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:12.587936 15401 hybrid_clock.cc:648] HybridClock initialized: now 1790607792587894 us; error 26 us; skew 500 ppm
I20260928 15:03:12.588366 15401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:12 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:12.594980 15401 init.cc:377] Logged in from keytab as oryx/127.9.12.254@KRBTEST.COM (short username oryx)
I20260928 15:03:12.596478 15401 webserver.cc:460] Webserver started at http://127.9.12.254:39575/ using document root <none> and password file <none>
I20260928 15:03:12.597155 15401 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:12.597316 15401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:12.597611 15401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:12.600538 15401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "9c2e21c0253c4818bc507daf0f09d796"
format_stamp: "Formatted at 2026-09-28 15:03:12 on dist-test-slave-dmhs"
server_key: "53bdc9b7c342548b64f88d365df84427"
server_key_iv: "ea0134405ec96e3983f0bf7be9892c10"
server_key_version: "encryptionkey@0"
I20260928 15:03:12.601293 15401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "9c2e21c0253c4818bc507daf0f09d796"
format_stamp: "Formatted at 2026-09-28 15:03:12 on dist-test-slave-dmhs"
server_key: "53bdc9b7c342548b64f88d365df84427"
server_key_iv: "ea0134405ec96e3983f0bf7be9892c10"
server_key_version: "encryptionkey@0"
I20260928 15:03:12.605998 15401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:03:12.609229 15416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:12.610443 15401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 15:03:12.610616 15401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "9c2e21c0253c4818bc507daf0f09d796"
format_stamp: "Formatted at 2026-09-28 15:03:12 on dist-test-slave-dmhs"
server_key: "53bdc9b7c342548b64f88d365df84427"
server_key_iv: "ea0134405ec96e3983f0bf7be9892c10"
server_key_version: "encryptionkey@0"
I20260928 15:03:12.610796 15401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:12.652899 15401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:12.657335 15401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:12.657626 15401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:12.675637 15401 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:40395
I20260928 15:03:12.675676 15467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:40395 every 8 connection(s)
I20260928 15:03:12.677047 15401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:12.679158 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15401
I20260928 15:03:12.679369 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:12.680212 9267 external_mini_cluster.cc:1428] Setting key 7997e39de9687ea14ed2a71c77d26e0d
I20260928 15:03:12.681587 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:03:12 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:12.695092 15468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Bootstrap starting.
I20260928 15:03:12.699153 15468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:12.700412 15468 log.cc:824] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:12.703251 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:12.682236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.9.12.254:40395)
0928 15:03:12.682923 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:12.682937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:12.682992 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:03:12.684986 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:12.685015 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:12.685517 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:12.685940 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:12.687206 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:12.689261 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:12.690435 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:12.691430 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:12.691608 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:12.695515 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:12.695524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:12.695541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:12.695584 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:12.699411 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:12.700048 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:12.700062 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:12.700072 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:12.700289 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:12.700672 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:12.700677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:12.700681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:12.701177 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:12.701523 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:12.701914 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 15:03:12.702312 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":287,"threads_started":1}
I20260928 15:03:12.703272 15468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: No bootstrap required, opened a new log
I20260928 15:03:12.710623 15468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } }
I20260928 15:03:12.710911 15468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:12.710973 15468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2e21c0253c4818bc507daf0f09d796, State: Initialized, Role: FOLLOWER
I20260928 15:03:12.711596 15468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } }
I20260928 15:03:12.711892 15468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:12.712052 15468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:12.712239 15468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:12.714344 15468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } }
I20260928 15:03:12.714871 15468 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2e21c0253c4818bc507daf0f09d796; no voters:
I20260928 15:03:12.715427 15468 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:12.715579 15473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:12.716307 15473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 LEADER]: Becoming Leader. State: Replica: 9c2e21c0253c4818bc507daf0f09d796, State: Running, Role: LEADER
I20260928 15:03:12.716912 15473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } }
I20260928 15:03:12.717737 15468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:12.719784 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } }
I20260928 15:03:12.720189 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:12.725384 15475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2e21c0253c4818bc507daf0f09d796. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } }
I20260928 15:03:12.725622 15475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:12.728941 15482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:12.733770 15482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:12.741463 15482 catalog_manager.cc:1347] Generated new cluster ID: 00cddc8856314a5f89a116473f4c4dff
I20260928 15:03:12.741645 15482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:12.754684 15482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:12.755934 15482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:12.767030 15482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Generated new TSK 0
I20260928 15:03:12.767876 15482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab.
Entry for principal oryx/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab.
I20260928 15:03:12.813426 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40395
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:13.119498 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:13.119740 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:13.119925 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:13.126929 15494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:13.127095 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:13.127293 15494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:13.137037 15494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:13.138332 15494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:13.139628 15494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:13.152627 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:13.152607 15500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:13.152699 15499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:13.153470 15494 server_base.cc:1034] running on GCE node
I20260928 15:03:13.154043 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:13.154958 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:13.156189 15494 hybrid_clock.cc:648] HybridClock initialized: now 1790607793156150 us; error 57 us; skew 500 ppm
I20260928 15:03:13.156563 15494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:13 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607793, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:13.159576 15494 init.cc:377] Logged in from keytab as oryx/127.9.12.193@KRBTEST.COM (short username oryx)
I20260928 15:03:13.161065 15494 webserver.cc:460] Webserver started at http://127.9.12.193:44771/ using document root <none> and password file <none>
I20260928 15:03:13.161682 15494 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:13.161777 15494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:13.162021 15494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:13.164806 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "a591c02989be4f2c96c8bb60ee820a66"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "a89241005dfc3bc0bbf3b3c701220243"
server_key_iv: "fcb6e6d3ee1335eb2c01d5017fa61ef5"
server_key_version: "encryptionkey@0"
I20260928 15:03:13.165485 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "a591c02989be4f2c96c8bb60ee820a66"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "a89241005dfc3bc0bbf3b3c701220243"
server_key_iv: "fcb6e6d3ee1335eb2c01d5017fa61ef5"
server_key_version: "encryptionkey@0"
I20260928 15:03:13.170599 15494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:03:13.173827 15509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:13.175069 15494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:13.175233 15494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "a591c02989be4f2c96c8bb60ee820a66"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "a89241005dfc3bc0bbf3b3c701220243"
server_key_iv: "fcb6e6d3ee1335eb2c01d5017fa61ef5"
server_key_version: "encryptionkey@0"
I20260928 15:03:13.175436 15494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:13.191071 15494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:13.194702 15494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:13.194998 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:13.195892 15494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:13.197225 15494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:13.197327 15494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:13.197491 15494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:13.197569 15494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:13.230540 15494 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:37065
I20260928 15:03:13.230587 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:37065 every 8 connection(s)
I20260928 15:03:13.231755 15494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:13.241734 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15494
I20260928 15:03:13.241940 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:13.242774 9267 external_mini_cluster.cc:1428] Setting key 82b86b2a77d611ea91d999ed2b082869
Sep 28 15:03:13 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607793, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:13.252646 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:13.234455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:37991 (local address 127.9.12.254:40395)
0928 15:03:13.234865 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:13.234869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:13.235845 (+ 976us) server_negotiation.cc:407] Connection header received
0928 15:03:13.237632 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:13.237637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:13.237724 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:13.237858 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:13.239214 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.239922 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.241386 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.241755 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.241887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:13.246302 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:13.246306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:13.246310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:13.246339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:13.249186 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:13.250105 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:13.250109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:13.250111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:13.250221 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:13.250688 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:13.250691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:13.250693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:13.250969 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:13.251122 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:13.251849 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 15:03:13.252140 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
I20260928 15:03:13.253290 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:13.235028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:37991)
0928 15:03:13.235703 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:13.235739 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:13.237381 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:13.238091 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:13.238098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:13.238553 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:13.239050 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:13.239066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.240081 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.240084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:13.241248 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:13.241258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.242037 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.242047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:13.242352 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:13.243098 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:13.243125 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:13.246099 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:13.249324 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:13.249336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:13.249355 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:13.249987 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:13.250320 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:13.250327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:13.250333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:13.250580 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:13.251100 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:13.251111 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:13.251460 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 15:03:13.251689 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 15:03:13.252104 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":277,"threads_started":1}
I20260928 15:03:13.254907 15622 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40395
I20260928 15:03:13.255213 15622 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:13.255975 15622 heartbeater.cc:511] Master 127.9.12.254:40395 requested a full tablet report, sending...
I20260928 15:03:13.258121 15433 ts_manager.cc:194] Registered new tserver with Master: a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:13.259536 15433 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:37991
WARNING: no policy specified for oryx/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab.
Entry for principal oryx/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab.
I20260928 15:03:13.289721 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40395
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:13.607501 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:13.607797 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:13.607951 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:13.615227 15628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:13.615398 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:13.615566 15628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:13.625504 15628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:13.626792 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:13.628161 15628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:13.641206 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:13.641248 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:13.641276 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:13.642345 15628 server_base.cc:1034] running on GCE node
I20260928 15:03:13.642957 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:13.643841 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:13.645068 15628 hybrid_clock.cc:648] HybridClock initialized: now 1790607793645012 us; error 58 us; skew 500 ppm
I20260928 15:03:13.645411 15628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:13 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607793, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:13.648488 15628 init.cc:377] Logged in from keytab as oryx/127.9.12.194@KRBTEST.COM (short username oryx)
I20260928 15:03:13.649761 15628 webserver.cc:460] Webserver started at http://127.9.12.194:35277/ using document root <none> and password file <none>
I20260928 15:03:13.650369 15628 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:13.650461 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:13.650686 15628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:13.653507 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "89b2cbaffa5045f79052d8d1f710a76a"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "312de9fcaccdd8a77c7e4d9005a4212a"
server_key_iv: "36349b2fd5220aebd8f85777d4a0457b"
server_key_version: "encryptionkey@0"
I20260928 15:03:13.654161 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "89b2cbaffa5045f79052d8d1f710a76a"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "312de9fcaccdd8a77c7e4d9005a4212a"
server_key_iv: "36349b2fd5220aebd8f85777d4a0457b"
server_key_version: "encryptionkey@0"
I20260928 15:03:13.659157 15628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 15:03:13.662456 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:13.663721 15628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:03:13.663882 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "89b2cbaffa5045f79052d8d1f710a76a"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "312de9fcaccdd8a77c7e4d9005a4212a"
server_key_iv: "36349b2fd5220aebd8f85777d4a0457b"
server_key_version: "encryptionkey@0"
I20260928 15:03:13.664084 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:13.688737 15628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:13.692521 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:13.692816 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:13.693753 15628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:13.695045 15628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:13.695127 15628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:13.695287 15628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:13.695338 15628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:13.728746 15628 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:34177
I20260928 15:03:13.728785 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:34177 every 8 connection(s)
I20260928 15:03:13.729983 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:13.739435 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15628
I20260928 15:03:13.739609 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:13.740172 9267 external_mini_cluster.cc:1428] Setting key 1b07c3d686e7f28d565467ba2f8e0b00
Sep 28 15:03:13 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607793, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:13.750347 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:13.732221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:39023 (local address 127.9.12.254:40395)
0928 15:03:13.732431 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:13.732437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:13.733704 (+ 1267us) server_negotiation.cc:407] Connection header received
0928 15:03:13.735335 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:13.735339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:13.735403 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:13.735504 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:13.736922 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.737643 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.739109 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.739761 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.739888 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:13.743322 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:13.743331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:13.743340 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:13.743384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:13.747062 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:13.747855 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:13.747864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:13.747870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:13.748036 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:13.748503 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:13.748511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:13.748516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:13.748913 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:13.749070 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:13.749694 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 15:03:13.749960 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:03:13.750955 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:13.732792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:39023)
0928 15:03:13.733468 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:13.733514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:13.735101 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:13.735683 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:13.735690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:13.736154 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:13.736781 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:13.736803 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.737772 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.737776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:13.738976 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:13.738988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:13.739952 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:13.739957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:13.740140 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:13.740901 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:13.740932 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:13.743115 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:13.747197 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:13.747203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:13.747215 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:13.747711 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:13.748157 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:13.748160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:13.748163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:13.748382 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:13.749032 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:13.749037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:13.749314 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 15:03:13.749528 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 15:03:13.749868 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
I20260928 15:03:13.752350 15756 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40395
I20260928 15:03:13.752648 15756 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:13.753312 15756 heartbeater.cc:511] Master 127.9.12.254:40395 requested a full tablet report, sending...
I20260928 15:03:13.754802 15433 ts_manager.cc:194] Registered new tserver with Master: 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177)
I20260928 15:03:13.756033 15433 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:39023
WARNING: no policy specified for oryx/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab.
Entry for principal oryx/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab.
I20260928 15:03:13.787688 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40395
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:14.097395 15761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:14.097664 15761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:14.097760 15761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:14.104921 15761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:14.105062 15761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:14.105247 15761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:14.114290 15761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:14.115562 15761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:14.116839 15761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:14.130177 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.130301 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.130333 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:14.131163 15761 server_base.cc:1034] running on GCE node
I20260928 15:03:14.131752 15761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:14.132771 15761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:14.133966 15761 hybrid_clock.cc:648] HybridClock initialized: now 1790607794133925 us; error 29 us; skew 500 ppm
I20260928 15:03:14.134284 15761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:14.137372 15761 init.cc:377] Logged in from keytab as oryx/127.9.12.195@KRBTEST.COM (short username oryx)
I20260928 15:03:14.138860 15761 webserver.cc:460] Webserver started at http://127.9.12.195:36281/ using document root <none> and password file <none>
I20260928 15:03:14.139551 15761 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:14.139690 15761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:14.139990 15761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:14.142906 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "276bf8b78da443fe9cbbdf6c6c172769"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "616d7142a7026111016f067e767b8a85"
server_key_iv: "c3118fdb4245d9a58d2e5c852d8f2892"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.143702 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "276bf8b78da443fe9cbbdf6c6c172769"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "616d7142a7026111016f067e767b8a85"
server_key_iv: "c3118fdb4245d9a58d2e5c852d8f2892"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.148988 15761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260928 15:03:14.152510 15776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:14.153820 15761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:03:14.153990 15761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "276bf8b78da443fe9cbbdf6c6c172769"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "616d7142a7026111016f067e767b8a85"
server_key_iv: "c3118fdb4245d9a58d2e5c852d8f2892"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.154182 15761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:14.172714 15761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:14.176527 15761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:14.176798 15761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:14.177749 15761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:14.178995 15761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:14.179095 15761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:14.179224 15761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:14.179306 15761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:14.212669 15761 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:40671
I20260928 15:03:14.212751 15888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:40671 every 8 connection(s)
I20260928 15:03:14.213999 15761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:14.216720 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15761
I20260928 15:03:14.216907 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:14.217518 9267 external_mini_cluster.cc:1428] Setting key 4b475b688d284b3b2b452c545c51a0af
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:14.234349 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.216734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:36105 (local address 127.9.12.254:40395)
0928 15:03:14.217013 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.217019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.218335 (+ 1316us) server_negotiation.cc:407] Connection header received
0928 15:03:14.220517 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.220522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.220603 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.220712 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.222046 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.222713 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.224170 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.224721 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.224850 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.228125 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.228131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.228136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.228189 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.230902 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.231822 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.231830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.231836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.231980 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.232527 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.232531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.232534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.232889 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.233035 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.233710 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.233966 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 15:03:14.235095 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.217256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:36105)
0928 15:03:14.218157 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.218208 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.220222 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.220935 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.220942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.221394 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.221891 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.221906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.222854 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.222859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.224036 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.224047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.224952 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.224961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.225204 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.225780 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.225805 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.227924 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.231047 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.231055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.231070 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.231638 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.232100 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.232104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.232107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.232368 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.233096 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.233101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.233382 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.233607 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.233997 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":342,"threads_started":1}
I20260928 15:03:14.236464 15889 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40395
I20260928 15:03:14.236788 15889 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:14.237455 15889 heartbeater.cc:511] Master 127.9.12.254:40395 requested a full tablet report, sending...
I20260928 15:03:14.238757 15433 ts_manager.cc:194] Registered new tserver with Master: 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195:40671)
I20260928 15:03:14.239459 15433 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:36105
I20260928 15:03:14.244992 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:14.262405 15622 heartbeater.cc:503] Master 127.9.12.254:40395 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.9.12.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.253.keytab.
Entry for principal oryx/127.9.12.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.253.keytab.
I20260928 15:03:14.305826 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:46699
--webserver_interface=127.9.12.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.253.keytab
--principal=oryx/127.9.12.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40395,127.9.12.253:46699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:14.628551 15894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:14.628831 15894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:14.629009 15894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:14.636241 15894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:14.636359 15894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:14.636404 15894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:14.636453 15894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:14.636544 15894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:14.646615 15894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40395,127.9.12.253:46699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.253.keytab
--principal=oryx/127.9.12.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.12.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
W20260928 15:03:14.647012 15894 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.12.254:40395,127.9.12.253:46699'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 15:03:14.648067 15894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:14.649464 15894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:14.660800 15899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.660702 15902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.660833 15900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:14.661947 15894 server_base.cc:1034] running on GCE node
I20260928 15:03:14.662663 15894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:14.664026 15894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:14.665267 15894 hybrid_clock.cc:648] HybridClock initialized: now 1790607794665179 us; error 95 us; skew 500 ppm
I20260928 15:03:14.665603 15894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:14.668900 15894 init.cc:377] Logged in from keytab as oryx/127.9.12.253@KRBTEST.COM (short username oryx)
I20260928 15:03:14.670199 15894 webserver.cc:460] Webserver started at http://127.9.12.253:46655/ using document root <none> and password file <none>
I20260928 15:03:14.670836 15894 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:14.670920 15894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:14.671166 15894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:14.674005 15894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/instance:
uuid: "e4144ee7a7954d74a6ecef86288072df"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "20ffb4635bb9afae51769ec5be9c9fc4"
server_key_iv: "e9b282e2421817b31352b082eecede02"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.674818 15894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal/instance:
uuid: "e4144ee7a7954d74a6ecef86288072df"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "20ffb4635bb9afae51769ec5be9c9fc4"
server_key_iv: "e9b282e2421817b31352b082eecede02"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.679929 15894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:03:14.683238 15909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:14.684525 15894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:14.684698 15894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
uuid: "e4144ee7a7954d74a6ecef86288072df"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "20ffb4635bb9afae51769ec5be9c9fc4"
server_key_iv: "e9b282e2421817b31352b082eecede02"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.684918 15894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:14.695976 15894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:14.700191 15894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:14.700501 15894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:14.718734 15894 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.253:46699
I20260928 15:03:14.718823 15960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.253:46699 every 8 connection(s)
I20260928 15:03:14.720427 15894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
I20260928 15:03:14.724380 15961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:14.726001 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15894
I20260928 15:03:14.734766 15961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.253@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:14.749972 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.728159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.9.12.253:46699)
0928 15:03:14.728991 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.729002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.729041 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:03:14.731254 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.731277 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.731684 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.732171 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.733681 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.735081 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.736504 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.737442 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.738350 (+ 908us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.741902 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.741913 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.741927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.741970 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.745278 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.746069 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.746077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.746085 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.746271 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.746628 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.746637 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.746643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.747351 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.747658 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.748077 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.748516 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":349,"threads_started":1}
I20260928 15:03:14.753216 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.737290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:35720)
0928 15:03:14.738086 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.738117 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.738282 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.738713 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.738719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.739338 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.739582 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.739590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.740696 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.740700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.741778 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.741790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.742781 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.742789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.742986 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.743559 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.743584 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.745823 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.749509 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.749518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.749522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.749977 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.751316 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.751319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.751322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.751449 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.752144 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.752150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.752354 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.752542 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.752892 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":470,"threads_started":1}
I20260928 15:03:14.753257 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.737307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.9.12.254:40395)
0928 15:03:14.737775 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.737783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.738207 (+ 424us) server_negotiation.cc:407] Connection header received
0928 15:03:14.738434 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.738439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.738528 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.738654 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.739728 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.740556 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.741918 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.742561 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.742718 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.746022 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.746030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.746034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.746073 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.749380 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.750096 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.750102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.750105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.751197 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.751583 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.751590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.751593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.752016 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.752162 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.752702 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.752944 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":248,"threads_started":1}
I20260928 15:03:14.754806 15961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:03:14.758558 15756 heartbeater.cc:503] Master 127.9.12.254:40395 was elected leader, sending a full tablet report...
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.253@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
I20260928 15:03:14.767000 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.755607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50712)
0928 15:03:14.755812 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.755830 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.755959 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.756244 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.756248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.756629 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.756748 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.756756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.757761 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.757767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.758696 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.758710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.759483 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.759488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.759748 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.760343 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.760360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.762132 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.764727 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.764734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.764738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.765080 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.765420 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.765426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.765429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.765552 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.766056 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.766060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.766204 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.766333 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.766617 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":20}
I20260928 15:03:14.767096 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.755659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.9.12.253:46699)
0928 15:03:14.755806 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.755812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.755901 (+ 89us) server_negotiation.cc:407] Connection header received
0928 15:03:14.756067 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.756072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.756136 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.756229 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.756875 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.757631 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.758828 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.759345 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.759474 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.762289 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.762294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.762298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.762335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.764604 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.765187 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.765195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.765199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.765317 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.765669 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.765676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.765680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.765947 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.766097 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.766409 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.766735 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":49}
I20260928 15:03:14.771821 15961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Bootstrap starting.
I20260928 15:03:14.774884 15961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:14.775906 15961 log.cc:824] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:14.778429 15961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: No bootstrap required, opened a new log
I20260928 15:03:14.785198 15961 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } }
I20260928 15:03:14.785550 15961 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:14.785667 15961 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4144ee7a7954d74a6ecef86288072df, State: Initialized, Role: FOLLOWER
I20260928 15:03:14.786245 15961 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } }
I20260928 15:03:14.787119 15968 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } } }
I20260928 15:03:14.787524 15968 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:14.788199 15961 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:03:14.796151 15979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:14.796294 15979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:14.798811 15894 master_runner.cc:428] Detected that this master e4144ee7a7954d74a6ecef86288072df is joining an existing cluster
I20260928 15:03:14.798928 15894 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.12.253:46699
I20260928 15:03:14.827481 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.802715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.9.12.254:40395)
0928 15:03:14.802907 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.802916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.810415 (+ 7499us) server_negotiation.cc:407] Connection header received
0928 15:03:14.810876 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.810883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.810977 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.811046 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.812646 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.813746 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.816150 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.816950 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.817057 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.820053 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.820073 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.820082 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.820126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.823721 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.824602 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.824610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.824617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.824779 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.825325 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.825333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.825339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.825718 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.825864 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.826664 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.826992 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:03:14.828682 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.802802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:35736)
0928 15:03:14.810312 (+ 7510us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.810332 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.810727 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.811269 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.811275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.812022 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.812471 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.812487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.813986 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.813994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.815980 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.815997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.817376 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.817384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.817612 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.818313 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.818338 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.819845 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.823871 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.823878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.823884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.824455 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.824937 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.824944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.824950 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.825166 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.825966 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.825974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.826304 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.827754 (+ 1450us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.828124 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7338,"mutex_wait_us":27,"thread_start_us":15178,"threads_started":1}
WARNING: no policy specified for oryx/127.9.12.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.252@KRBTEST.COM" created.
I20260928 15:03:14.835409 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.803033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.9.12.253:46699)
0928 15:03:14.803223 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.803228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.817957 (+ 14729us) server_negotiation.cc:407] Connection header received
0928 15:03:14.818456 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.818461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.818533 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.818604 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.820210 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.821576 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.824749 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.825472 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.825580 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.828598 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.828604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.828609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.828656 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.832458 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.833249 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.833255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.833258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.833412 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.833819 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.833824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.833827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.834187 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.834355 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.834841 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.835063 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:03:14.836566 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.803669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50726)
0928 15:03:14.817820 (+ 14151us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.817839 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.818284 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.818874 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.818879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.819539 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.820065 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.820078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.821809 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.821814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.824604 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.824619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.825842 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.825848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.826130 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.826753 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.826775 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.828415 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.832602 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.832607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.832611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.833123 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.833559 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.833565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.833570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.833717 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.834431 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.834436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.834583 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.835847 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.836113 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14018,"mutex_wait_us":74,"thread_start_us":6755,"threads_started":1}
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.254@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
I20260928 15:03:14.856582 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.837474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.9.12.253:46699)
0928 15:03:14.837618 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.837628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.838276 (+ 648us) server_negotiation.cc:407] Connection header received
0928 15:03:14.838650 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.838656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.838763 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.838830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.840695 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.841748 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.843604 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.844481 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.844622 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.848931 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.848938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.848944 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.848987 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.853020 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.854028 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.854034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.854037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.854191 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.854694 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.854700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.854704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.855141 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.855298 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.856008 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.856191 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:03:14.856734 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.835139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50728)
0928 15:03:14.838148 (+ 3009us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.838198 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.838431 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.839106 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.839114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.840186 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.840532 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.840545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.841924 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.841930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.843450 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.843465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.844655 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.844662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.844884 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.845626 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.845650 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.848715 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.853196 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.853216 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.853226 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.853872 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.854356 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.854364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.854369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.854555 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.855383 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.855392 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.855666 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.855901 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.856287 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2768,"thread_start_us":2596,"threads_started":1}
I20260928 15:03:14.858882 15433 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.12.253:46699: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4144ee7a7954d74a6ecef86288072df"
member_type: NON_VOTER
last_known_addr {
host: "127.9.12.253"
port: 46699
}
attrs {
promote: true
}
}
dest_uuid: "9c2e21c0253c4818bc507daf0f09d796"
cas_config_opid_index: -1
I20260928 15:03:14.859977 15433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.252.keytab.
Entry for principal oryx/127.9.12.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.252.keytab.
I20260928 15:03:14.863965 15936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:14.864702 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:45969
--webserver_interface=127.9.12.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.252.keytab
--principal=oryx/127.9.12.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:14.868198 15936 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Refusing update from remote peer 9c2e21c0253c4818bc507daf0f09d796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 15:03:14.869699 15992 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e4144ee7a7954d74a6ecef86288072df (127.9.12.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } }
I20260928 15:03:14.874877 15991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:14.877005 15992 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.12.253:46699
I20260928 15:03:14.877804 15894 master.cc:537] Master@127.9.12.253:46699 shutting down...
I20260928 15:03:14.877867 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } }
I20260928 15:03:14.878219 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
W20260928 15:03:14.882285 15418 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 15:03:14.886895 15418 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 -> Peer e4144ee7a7954d74a6ecef86288072df (127.9.12.253:46699): Couldn't send request to peer e4144ee7a7954d74a6ecef86288072df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:03:14.891047 15894 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:14.891465 15894 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:14.891657 15894 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: stopping tablet replica
I20260928 15:03:14.898783 15894 master.cc:552] Master@127.9.12.253:46699 shutdown complete.
I20260928 15:03:14.898874 15894 master_runner.cc:305] Clearing existing system tablet
I20260928 15:03:14.899353 15894 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:14.899456 15894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:14.902781 15894 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 15:03:14.904443 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:14.904783 15894 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:14.905179 15894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 15:03:14.905311 15894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
uuid: "e4144ee7a7954d74a6ecef86288072df"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "20ffb4635bb9afae51769ec5be9c9fc4"
server_key_iv: "e9b282e2421817b31352b082eecede02"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.910342 15894 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:14.917995 15894 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:03:14.918316 15894 log.cc:1196] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 15:03:14.918839 15894 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Deleting consensus metadata
I20260928 15:03:14.920552 15894 master_runner.cc:315] Copying system tablet from 127.9.12.254:40395
I20260928 15:03:14.923187 15894 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: tablet copy: Beginning tablet copy session from remote peer at address 127.9.12.254:40395
I20260928 15:03:14.943511 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.924033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.9.12.254:40395)
0928 15:03:14.924217 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.924223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.928890 (+ 4667us) server_negotiation.cc:407] Connection header received
0928 15:03:14.929106 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.929112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.929200 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.929314 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.930471 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.931633 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.933625 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.934284 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.934437 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.936833 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.936840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.936845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.936884 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.940317 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.941244 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.941249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.941253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.941399 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.941855 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.941860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.941864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.942229 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.942388 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.942882 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.943110 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:03:14.943554 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.924025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:35752)
0928 15:03:14.928733 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:14.928759 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:14.928934 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:14.929365 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:14.929370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:14.930000 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:14.930295 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.930307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.931800 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.931806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.933471 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:14.933488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.934563 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.934569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:14.934747 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.935482 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:14.935510 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:14.936620 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:14.940452 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.940458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.940463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.941095 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.941543 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:14.941550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:14.941555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:14.941732 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:14.942376 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:14.942381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:14.942578 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 15:03:14.942727 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 15:03:14.943107 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4548,"thread_start_us":4433,"threads_started":1}
I20260928 15:03:14.944792 15453 tablet_copy_service.cc:140] P 9c2e21c0253c4818bc507daf0f09d796: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4144ee7a7954d74a6ecef86288072df ({username='oryx', principal='oryx/127.9.12.253@KRBTEST.COM'} at 127.0.0.1:35752)
I20260928 15:03:14.945070 15453 tablet_copy_service.cc:161] P 9c2e21c0253c4818bc507daf0f09d796: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4144ee7a7954d74a6ecef86288072df at {username='oryx', principal='oryx/127.9.12.253@KRBTEST.COM'} at 127.0.0.1:35752: session id = e4144ee7a7954d74a6ecef86288072df-00000000000000000000000000000000
I20260928 15:03:14.949642 15453 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:03:14.951786 15894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:03:14.959079 15894 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: tablet copy: Starting download of 0 data blocks...
I20260928 15:03:14.959414 15894 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: tablet copy: Starting download of 1 WAL segments...
I20260928 15:03:14.962178 15894 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:03:14.966522 15453 tablet_copy_service.cc:342] P 9c2e21c0253c4818bc507daf0f09d796: Request end of tablet copy session e4144ee7a7954d74a6ecef86288072df-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.12.253@KRBTEST.COM'} at 127.0.0.1:35752
I20260928 15:03:14.966723 15453 tablet_copy_service.cc:433] P 9c2e21c0253c4818bc507daf0f09d796: ending tablet copy session e4144ee7a7954d74a6ecef86288072df-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4144ee7a7954d74a6ecef86288072df
I20260928 15:03:14.979328 15894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:14.982012 16006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.986687 16007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.991218 16009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:14.991559 15894 server_base.cc:1034] running on GCE node
I20260928 15:03:14.991868 15894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:14.992523 15894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:14.993789 15894 hybrid_clock.cc:648] HybridClock initialized: now 1790607794993702 us; error 48 us; skew 500 ppm
I20260928 15:03:14.994048 15894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:14 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:14.996362 15894 init.cc:377] Logged in from keytab as oryx/127.9.12.253@KRBTEST.COM (short username oryx)
I20260928 15:03:14.999225 15894 webserver.cc:460] Webserver started at http://127.9.12.253:37985/ using document root <none> and password file <none>
I20260928 15:03:14.999612 15894 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:14.999723 15894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.002806 15894 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:15.005132 16016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.005559 15894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:03:15.005751 15894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
uuid: "e4144ee7a7954d74a6ecef86288072df"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "20ffb4635bb9afae51769ec5be9c9fc4"
server_key_iv: "e9b282e2421817b31352b082eecede02"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.005988 15894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.028638 15894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.032701 15894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.052584 15894 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.253:46699
I20260928 15:03:15.052621 16067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.253:46699 every 8 connection(s)
I20260928 15:03:15.053726 15894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
I20260928 15:03:15.058941 16068 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:03:15.062444 16068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Bootstrap starting.
I20260928 15:03:15.078501 16068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:03:15.079064 16068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: Bootstrap complete.
I20260928 15:03:15.080252 16068 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } }
I20260928 15:03:15.080585 16068 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4144ee7a7954d74a6ecef86288072df, State: Initialized, Role: LEARNER
I20260928 15:03:15.080894 16068 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } }
I20260928 15:03:15.082093 16068 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:15.081871 16071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } }
I20260928 15:03:15.082413 16071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:15.092124 16082 catalog_manager.cc:1259] Loaded cluster ID: 00cddc8856314a5f89a116473f4c4dff
I20260928 15:03:15.092242 16082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: loading cluster ID for follower catalog manager: success
I20260928 15:03:15.095747 16082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: acquiring CA information for follower catalog manager: success
I20260928 15:03:15.098369 16082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:03:15.241709 15889 heartbeater.cc:503] Master 127.9.12.254:40395 was elected leader, sending a full tablet report...
W20260928 15:03:15.288923 15994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:15.289259 15994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:15.289461 15994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:15.297466 15994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:15.297603 15994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:15.297708 15994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:15.297796 15994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:15.297860 15994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:15.308076 15994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.252.keytab
--principal=oryx/127.9.12.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:45969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.12.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:15.309442 15994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:15.310721 15994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.322866 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.322983 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.322944 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.333295 15994 server_base.cc:1034] running on GCE node
I20260928 15:03:15.334447 15994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.336620 15994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.337850 15994 hybrid_clock.cc:648] HybridClock initialized: now 1790607795337803 us; error 46 us; skew 500 ppm
I20260928 15:03:15.338225 15994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.341389 15994 init.cc:377] Logged in from keytab as oryx/127.9.12.252@KRBTEST.COM (short username oryx)
I20260928 15:03:15.342967 15994 webserver.cc:460] Webserver started at http://127.9.12.252:34479/ using document root <none> and password file <none>
I20260928 15:03:15.343663 15994 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.343757 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.344003 15994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:15.346998 15994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/instance:
uuid: "85712a8afd144249bf425b5e672ba007"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-dmhs"
server_key: "ad3abe49900e27dcafd88e021c444d28"
server_key_iv: "67c0da536398535e8d40a1664c2f51ca"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.347792 15994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal/instance:
uuid: "85712a8afd144249bf425b5e672ba007"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-dmhs"
server_key: "ad3abe49900e27dcafd88e021c444d28"
server_key_iv: "67c0da536398535e8d40a1664c2f51ca"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.353012 15994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:03:15.356362 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.357656 15994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:03:15.357831 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
uuid: "85712a8afd144249bf425b5e672ba007"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-dmhs"
server_key: "ad3abe49900e27dcafd88e021c444d28"
server_key_iv: "67c0da536398535e8d40a1664c2f51ca"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.358026 15994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.376667 15994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.380714 15994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:15.380995 15994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.402376 15994 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.252:45969
I20260928 15:03:15.402384 16148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.252:45969 every 8 connection(s)
I20260928 15:03:15.403790 15994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
I20260928 15:03:15.407518 16149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:15.413131 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 15994
I20260928 15:03:15.414057 16149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.252@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.252@KRBTEST.COM
I20260928 15:03:15.435204 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.416934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.9.12.254:40395)
0928 15:03:15.417132 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.417138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.418336 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 15:03:15.418876 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.418885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.418971 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.419072 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.421313 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.422445 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.424652 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.425036 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.425128 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.429008 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.429016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.429024 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.429088 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.432004 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.432959 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.432989 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.432993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.433119 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.433569 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.433572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.433574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.433825 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.433967 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.434636 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.434870 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:03:15.436427 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.416884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:35758)
0928 15:03:15.418183 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.418227 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.418647 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.419146 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.419152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.420221 (+ 1069us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.421130 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.421173 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.422627 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.422631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.424520 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.424535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.425266 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.425273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.425442 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.426221 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.426242 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.428821 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.432127 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.432135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.432144 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.432843 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.433226 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.433231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.433234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.433462 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.433958 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.433966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.434270 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.434534 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.434959 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":173,"thread_start_us":854,"threads_started":1}
I20260928 15:03:15.436713 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.415459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.9.12.252:45969)
0928 15:03:15.416203 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.416212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.416248 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:03:15.418637 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.418658 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.418922 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.419114 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.421780 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.423582 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.425248 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.425955 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.426104 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.430830 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.430838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.430849 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.430880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.433854 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.434511 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.434519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.434522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.434663 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.435069 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.435074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.435076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.435469 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.435776 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.436028 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.436237 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":355,"threads_started":1}
I20260928 15:03:15.438012 16149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.252@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
I20260928 15:03:15.453117 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.438672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50738)
0928 15:03:15.438888 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.438906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.438997 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.439751 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.439754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.440055 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.440171 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.440178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.441385 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.441388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.442250 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.442260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.442883 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.442887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.443029 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.443493 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.443509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.445403 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.449586 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.449592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.449605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.450373 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.451121 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.451129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.451135 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.451342 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.452071 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.452075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.452191 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.452421 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.452739 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:03:15.453187 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.438874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.9.12.253:46699)
0928 15:03:15.439441 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.439448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.439468 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:03:15.439531 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.439535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.439616 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.439722 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.440336 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.441267 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.442365 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.442773 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.442860 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.445650 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.445656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.445659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.445697 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.449425 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.450641 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.450648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.450654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.450952 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.451571 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.451576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.451579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.451893 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.451991 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.452525 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.452737 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":302,"threads_started":1}
I20260928 15:03:15.454561 16149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:03:15.455622 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.447252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50744)
0928 15:03:15.448137 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.448157 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.448250 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.449004 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.449010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.449041 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:15.449282 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.449295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.451618 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.451622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.452976 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.452990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.454741 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.454746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.454937 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.455008 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.455121 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.455264 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":657,"threads_started":1}
I20260928 15:03:15.456112 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.447502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.9.12.253:46699)
0928 15:03:15.448696 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.448701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.448714 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:15.448761 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.448767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.448829 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.448921 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:15.449441 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.451436 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.453127 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.454563 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.454706 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.454965 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.455142 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.455801 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":967,"threads_started":1}
I20260928 15:03:15.457002 16043 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 15:03:15.457562 16071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2e21c0253c4818bc507daf0f09d796. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } }
I20260928 15:03:15.457803 16071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: LEARNER
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.252@KRBTEST.COM for oryx/127.9.12.252@KRBTEST.COM
I20260928 15:03:15.467406 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.455469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.0.0.1:50726)
0928 15:03:15.455637 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.455655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.455807 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.456329 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.456334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.456956 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.457154 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.457162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.458303 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.458308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.459126 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.459136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.459793 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.459798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.460064 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.460605 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.460622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.462538 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.465356 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.465359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.465362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.465709 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.466012 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.466015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.466017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.466151 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.466564 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.466567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.466670 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.466784 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.467057 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":22}
I20260928 15:03:15.467448 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.455539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.9.12.252:45969)
0928 15:03:15.455724 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.455730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.455749 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:03:15.455920 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.455933 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.455994 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.456052 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.457286 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.458184 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.459248 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.459660 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.459785 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.462700 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.462704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.462707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.462752 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.465245 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.465800 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.465804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.465806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.465905 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.466222 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.466225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.466227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.466470 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.466571 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.466865 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.467155 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":77}
I20260928 15:03:15.472860 16149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Bootstrap starting.
I20260928 15:03:15.476008 16149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:15.477054 16149 log.cc:824] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:15.479799 16149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: No bootstrap required, opened a new log
I20260928 15:03:15.486903 16149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } }
I20260928 15:03:15.487226 16149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:15.487311 16149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85712a8afd144249bf425b5e672ba007, State: Initialized, Role: FOLLOWER
I20260928 15:03:15.487876 16149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } }
I20260928 15:03:15.488727 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } } }
I20260928 15:03:15.489053 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:15.489670 16149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:03:15.498759 16170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:15.498912 16170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:15.502851 15994 master_runner.cc:428] Detected that this master 85712a8afd144249bf425b5e672ba007 is joining an existing cluster
I20260928 15:03:15.502965 15994 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.12.252:45969
I20260928 15:03:15.531356 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.508037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.9.12.254:40395)
0928 15:03:15.508273 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.508301 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.514109 (+ 5808us) server_negotiation.cc:407] Connection header received
0928 15:03:15.514713 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.514720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.514808 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.514880 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.517346 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.518532 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.520773 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.521464 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.521587 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.524371 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.524378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.524383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.524423 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.528131 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.528943 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.528949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.528953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.529114 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.529655 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.529660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.529663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.530040 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.530169 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.530773 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.531007 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 15:03:15.532639 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.506872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:35760)
0928 15:03:15.514010 (+ 7138us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.514030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.514538 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.515092 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.515097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.515888 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.517171 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.517191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.518758 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.518763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.520582 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.520603 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.521881 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.521892 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.522079 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.522839 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.522860 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.524127 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.528310 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.528315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.528319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.528810 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.529258 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.529262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.529265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.529519 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.530245 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.530250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.530458 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.531885 (+ 1427us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.532165 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6983,"thread_start_us":11559,"threads_started":1}
I20260928 15:03:15.537262 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.507373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.9.12.252:45969)
0928 15:03:15.507629 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.507638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.519265 (+ 11627us) server_negotiation.cc:407] Connection header received
0928 15:03:15.519526 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.519534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.519614 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.519702 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.522391 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.523566 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.525550 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.526279 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.526419 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.529353 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.529360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.529365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.529406 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.533538 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.535196 (+ 1658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.535201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.535204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.535350 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.535719 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.535727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.535729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.535989 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.536163 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.536589 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.536874 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":72}
I20260928 15:03:15.538373 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.507024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.0.0.1:50742)
0928 15:03:15.519145 (+ 12121us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.519173 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.519368 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.519968 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.519975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.521757 (+ 1782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.522233 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.522243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.523800 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.523805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.525392 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.525405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.527074 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.527080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.527254 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.527885 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.527907 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.529179 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.534469 (+ 5290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.534473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.534477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.535078 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.535464 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.535469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.535473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.535604 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.536097 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.536104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.536297 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.537659 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.537924 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11901,"thread_start_us":14130,"threads_started":1}
I20260928 15:03:15.539149 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.506970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.9.12.253:46699)
0928 15:03:15.507144 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.507150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.521627 (+ 14477us) server_negotiation.cc:407] Connection header received
0928 15:03:15.521811 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.521818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.521908 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.522015 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.523466 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.524628 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.527049 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.527752 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.527862 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.530432 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.530438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.530442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.530482 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.534278 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.537007 (+ 2729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.537013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.537016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.537161 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.537619 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.537626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.537630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.537960 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.538079 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.538618 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.538830 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:03:15.540074 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.507379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50752)
0928 15:03:15.521515 (+ 14136us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.521529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.521662 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.522212 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.522217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.522907 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.523280 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.523295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.524809 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.524814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.526837 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.526850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.528167 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.528174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.528401 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.529007 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.529032 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.530206 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.534427 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.534432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.534436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.536871 (+ 2435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.537308 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.537313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.537316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.537467 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.538152 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.538157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.538350 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.539370 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.539641 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13966,"mutex_wait_us":1,"thread_start_us":6748,"threads_started":1}
I20260928 15:03:15.542351 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15494
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.254@KRBTEST.COM for oryx/127.9.12.252@KRBTEST.COM
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.193@KRBTEST.COM
I20260928 15:03:15.555955 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.539068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.0.0.1:50756)
0928 15:03:15.539367 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.539385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.539492 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.540498 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.540504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.540944 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.541141 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.541152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.542849 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.542855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.544578 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.544594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.546685 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.546689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.546811 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.547467 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.547484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.549791 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.553453 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.553461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.553465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.553858 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.554257 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.554262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.554265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.554396 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.554921 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.554926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.555128 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.555274 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.555575 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260928 15:03:15.555954 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.539679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.9.12.252:45969)
0928 15:03:15.540032 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.540037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.540050 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:15.540165 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.540171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.540344 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.541299 (+ 955us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.541857 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.542717 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.544749 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.545275 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.545417 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.549981 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.549988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.549992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.550033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.553305 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.553994 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.554000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.554004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.554132 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.554508 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.554514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.554517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.554800 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.554953 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.555359 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.555587 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260928 15:03:15.557876 15433 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.12.252:45969: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "85712a8afd144249bf425b5e672ba007"
member_type: NON_VOTER
last_known_addr {
host: "127.9.12.252"
port: 45969
}
attrs {
promote: true
}
}
dest_uuid: "9c2e21c0253c4818bc507daf0f09d796"
cas_config_opid_index: 5
I20260928 15:03:15.558923 15433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } }
I20260928 15:03:15.560737 16155 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85712a8afd144249bf425b5e672ba007 (127.9.12.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.561378 16043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 LEARNER]: Refusing update from remote peer 9c2e21c0253c4818bc507daf0f09d796: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:03:15.564684 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.543409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.9.12.193:37065)
0928 15:03:15.544835 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.544847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.544876 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:03:15.544943 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.544952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.545241 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.545422 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.546680 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.548334 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.550148 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.550789 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.550984 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.556890 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.556901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.556915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.556948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.560105 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.561785 (+ 1680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.561795 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.561798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.561938 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.562392 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.562400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.562406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.562950 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.563338 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.563681 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.564044 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":1065,"threads_started":1}
I20260928 15:03:15.565121 16124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:15.566325 15617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:15.569918 16124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Refusing update from remote peer 9c2e21c0253c4818bc507daf0f09d796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:03:15.575295 16183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 15:03:15.590809 16043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85712a8afd144249bf425b5e672ba007 (127.9.12.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.597273 16180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:15.597807 16155 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.12.252:45969
I20260928 15:03:15.598562 15994 master.cc:537] Master@127.9.12.252:45969 shutting down...
I20260928 15:03:15.599855 16071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.600143 16071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:15.602113 16155 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: attempting to promote NON_VOTER e4144ee7a7954d74a6ecef86288072df to VOTER
I20260928 15:03:15.602003 16183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.602458 16183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:15.604144 16155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } }
I20260928 15:03:15.606035 16181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: NON_VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: true } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.606644 16181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:15.608059 16043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 LEARNER]: Refusing update from remote peer 9c2e21c0253c4818bc507daf0f09d796: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 15:03:15.608814 16181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 15:03:15.611495 16042 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4144ee7a7954d74a6ecef86288072df (127.9.12.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.615664 16071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.615919 16071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:15.616688 16180 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4144ee7a7954d74a6ecef86288072df (127.9.12.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.621598 16183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.621891 16183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:15.622167 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.622479 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
W20260928 15:03:15.624625 15417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 -> Peer 85712a8afd144249bf425b5e672ba007 (127.9.12.252:45969): Couldn't send request to peer 85712a8afd144249bf425b5e672ba007. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20260928 15:03:15.633251 16103 tablet.cc:2369] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:15.635210 15994 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:15.635643 15994 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:15.635854 15994 tablet_replica.cc:330] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: stopping tablet replica
I20260928 15:03:15.652366 15994 master.cc:552] Master@127.9.12.252:45969 shutdown complete.
I20260928 15:03:15.652467 15994 master_runner.cc:305] Clearing existing system tablet
I20260928 15:03:15.652974 15994 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.653124 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.656133 15994 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 15:03:15.657753 16189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.658313 15994 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.658597 15994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:03:15.658730 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
uuid: "85712a8afd144249bf425b5e672ba007"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-dmhs"
server_key: "ad3abe49900e27dcafd88e021c444d28"
server_key_iv: "67c0da536398535e8d40a1664c2f51ca"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.663548 15994 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:15.670354 15994 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:03:15.670523 15994 log.cc:1196] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 15:03:15.670966 15994 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Deleting consensus metadata
I20260928 15:03:15.671204 15994 master_runner.cc:315] Copying system tablet from 127.9.12.254:40395
I20260928 15:03:15.700194 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15494
I20260928 15:03:15.700615 15994 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: tablet copy: Beginning tablet copy session from remote peer at address 127.9.12.254:40395
I20260928 15:03:15.720575 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.701433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.9.12.254:40395)
0928 15:03:15.701669 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.701676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.708840 (+ 7164us) server_negotiation.cc:407] Connection header received
0928 15:03:15.709086 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.709091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.709189 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.709286 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.710123 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.710796 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.712000 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.712495 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.712616 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.714656 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.714662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.714667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.714723 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.717700 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.718444 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.718450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.718453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.718610 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.719096 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.719100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.719104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.719368 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.719555 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.719964 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.720215 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 15:03:15.720656 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.701423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:35768)
0928 15:03:15.708728 (+ 7305us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.708747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.708949 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.709319 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.709324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.709763 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.709975 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.709987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.710927 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.710932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.711853 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.711863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.712648 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.712655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.712817 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.713466 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.713505 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.714448 (+ 943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.717890 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.717894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.717896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.718315 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.718718 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.718723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.718727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.718869 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.719512 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.719517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.719710 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.719862 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.720184 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7157,"thread_start_us":7021,"threads_started":1}
I20260928 15:03:15.721696 15453 tablet_copy_service.cc:140] P 9c2e21c0253c4818bc507daf0f09d796: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 85712a8afd144249bf425b5e672ba007 ({username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:35768)
I20260928 15:03:15.721875 15453 tablet_copy_service.cc:161] P 9c2e21c0253c4818bc507daf0f09d796: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 85712a8afd144249bf425b5e672ba007 at {username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:35768: session id = 85712a8afd144249bf425b5e672ba007-00000000000000000000000000000000
I20260928 15:03:15.724536 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:37065
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:40395
--webserver_port=44771
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:15.724839 15453 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:03:15.727006 15994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:03:15.733619 15994 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: tablet copy: Starting download of 0 data blocks...
I20260928 15:03:15.733896 15994 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: tablet copy: Starting download of 1 WAL segments...
I20260928 15:03:15.736129 15994 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:03:15.739151 15453 tablet_copy_service.cc:342] P 9c2e21c0253c4818bc507daf0f09d796: Request end of tablet copy session 85712a8afd144249bf425b5e672ba007-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:35768
I20260928 15:03:15.739310 15453 tablet_copy_service.cc:433] P 9c2e21c0253c4818bc507daf0f09d796: ending tablet copy session 85712a8afd144249bf425b5e672ba007-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 85712a8afd144249bf425b5e672ba007
I20260928 15:03:15.749634 15994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.752197 16197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.758215 16198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.762706 16200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.765206 15994 server_base.cc:1034] running on GCE node
I20260928 15:03:15.765558 15994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.766196 15994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.767519 15994 hybrid_clock.cc:648] HybridClock initialized: now 1790607795767475 us; error 32 us; skew 500 ppm
I20260928 15:03:15.767761 15994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:15 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.769769 15994 init.cc:377] Logged in from keytab as oryx/127.9.12.252@KRBTEST.COM (short username oryx)
I20260928 15:03:15.778206 15994 webserver.cc:460] Webserver started at http://127.9.12.252:38743/ using document root <none> and password file <none>
I20260928 15:03:15.778505 15994 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.778622 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.780915 15994 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:15.782557 16207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.782907 15994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:15.783149 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
uuid: "85712a8afd144249bf425b5e672ba007"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-dmhs"
server_key: "ad3abe49900e27dcafd88e021c444d28"
server_key_iv: "67c0da536398535e8d40a1664c2f51ca"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.783339 15994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.817337 15994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.821238 15994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.847020 15994 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.252:45969
I20260928 15:03:15.847070 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.252:45969 every 8 connection(s)
I20260928 15:03:15.847991 15994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
I20260928 15:03:15.855249 16259 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:03:15.861095 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Bootstrap starting.
I20260928 15:03:15.874517 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:03:15.875025 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Bootstrap complete.
I20260928 15:03:15.875953 16259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } }
I20260928 15:03:15.876217 16259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85712a8afd144249bf425b5e672ba007, State: Initialized, Role: LEARNER
I20260928 15:03:15.876499 16259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } }
I20260928 15:03:15.877302 16262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:15.877647 16259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:15.877766 16262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:15.893574 16273 catalog_manager.cc:1259] Loaded cluster ID: 00cddc8856314a5f89a116473f4c4dff
I20260928 15:03:15.893677 16273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: loading cluster ID for follower catalog manager: success
I20260928 15:03:15.897027 16273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: acquiring CA information for follower catalog manager: success
I20260928 15:03:15.899325 16273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:03:16.118148 16196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:16.118535 16196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:16.118672 16196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:16.126314 16196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:16.126408 16196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:16.126511 16196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:16.136065 16196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:37065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=44771
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:16.137320 16196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:16.138532 16196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:16.151580 16278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.151494 16281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.151538 16279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:16.152626 16196 server_base.cc:1034] running on GCE node
I20260928 15:03:16.153182 16196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:16.154008 16196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:16.155210 16196 hybrid_clock.cc:648] HybridClock initialized: now 1790607796155144 us; error 50 us; skew 500 ppm
I20260928 15:03:16.155486 16196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:16.158505 16196 init.cc:377] Logged in from keytab as oryx/127.9.12.193@KRBTEST.COM (short username oryx)
I20260928 15:03:16.160094 16196 webserver.cc:460] Webserver started at http://127.9.12.193:44771/ using document root <none> and password file <none>
I20260928 15:03:16.160835 16196 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:16.161002 16196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:16.166711 16196 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:03:16.169710 16288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.171026 16196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:16.171221 16196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "a591c02989be4f2c96c8bb60ee820a66"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "a89241005dfc3bc0bbf3b3c701220243"
server_key_iv: "fcb6e6d3ee1335eb2c01d5017fa61ef5"
server_key_version: "encryptionkey@0"
I20260928 15:03:16.172156 16196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:16.191491 16196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:16.196106 16196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:16.196420 16196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:16.197405 16196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:16.199584 16196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:16.199707 16196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.199816 16196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:16.199914 16196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.206629 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.198210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.0.0.1:50772)
0928 15:03:16.198770 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.198803 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.198975 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.199490 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.199508 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.199536 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:16.199767 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.199780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.201308 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.201313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.204021 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.204047 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.205641 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.205650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.205868 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.205910 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.206015 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.206212 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260928 15:03:16.207190 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.198367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.9.12.252:45969)
0928 15:03:16.199113 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.199120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.199144 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:03:16.199229 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.199234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.199326 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.199426 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:16.199919 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.201161 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.204199 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.205435 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.205556 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.205823 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.206075 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.206351 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":450,"threads_started":1}
I20260928 15:03:16.208349 16234 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 15:03:16.209177 16262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2e21c0253c4818bc507daf0f09d796. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:16.209522 16262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:16.247987 16196 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:37065
I20260928 15:03:16.248077 16403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:37065 every 8 connection(s)
I20260928 15:03:16.249457 16196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:16.252043 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 16196
I20260928 15:03:16.252440 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15628
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.252@KRBTEST.COM
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:16.293200 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.253616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51007 (local address 127.9.12.254:40395)
0928 15:03:16.269761 (+ 16145us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.269771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.274015 (+ 4244us) server_negotiation.cc:407] Connection header received
0928 15:03:16.274490 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.274498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.274589 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.274668 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.275950 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.277225 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.279910 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.280565 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.280667 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.285923 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.285929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.285935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.285982 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.290104 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.290982 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.290987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.290990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.291124 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.291585 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.291590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.291592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.291924 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.292040 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.292574 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.292800 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15975,"thread_start_us":15848,"threads_started":1}
I20260928 15:03:16.293591 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.253290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:45073 (local address 127.9.12.252:45969)
0928 15:03:16.253506 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.253512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.270461 (+ 16949us) server_negotiation.cc:407] Connection header received
0928 15:03:16.270640 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.270648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.270754 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.270872 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.274594 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.275742 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.279008 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.279803 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.279910 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.285675 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.285696 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.285701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.285741 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.290314 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.291203 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.291208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.291212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.291355 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.291845 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.291850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.291853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.292205 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.292370 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.292927 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.293199 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:03:16.294324 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.252947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.9.12.193:43081)
0928 15:03:16.262476 (+ 9529us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.262534 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.268827 (+ 6293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.271479 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.271486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.272705 (+ 1219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.273716 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.273748 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.275770 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.275779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.278431 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.278450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.279780 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.279786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.280078 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.281011 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.281055 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.284789 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:16.289537 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.289546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.289563 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.290150 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.290562 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.290569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.290575 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.290826 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.291555 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:16.291566 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:16.292038 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.292365 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.292888 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9198,"mutex_wait_us":39,"thread_start_us":16900,"threads_started":1}
I20260928 15:03:16.294636 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.252947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.9.12.193:45073)
0928 15:03:16.270305 (+ 17358us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.270331 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.270495 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.271443 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.271460 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.274077 (+ 2617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.274409 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.274422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.275945 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.275949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.278828 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.278841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.280218 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.280226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.280470 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.281170 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.281193 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.285464 (+ 4271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:16.290490 (+ 5026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.290495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.290499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.291078 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.291526 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.291531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.291536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.291682 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.292460 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:16.292465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:16.292619 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.293767 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.294051 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17147,"mutex_wait_us":508,"thread_start_us":20627,"threads_started":1}
I20260928 15:03:16.295481 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.252474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:43081 (local address 127.9.12.253:46699)
0928 15:03:16.271050 (+ 18576us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.271059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.271089 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:03:16.271161 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.271169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.271275 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.272438 (+ 1163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.273923 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.275487 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.278609 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.279388 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.279501 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.285217 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.285226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.285230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.285274 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.289366 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.290277 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.290282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.290285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.290419 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.290981 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.290985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.290989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.291327 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.291482 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.294826 (+ 3344us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.295074 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18398,"thread_start_us":18251,"threads_started":1}
I20260928 15:03:16.295521 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.253387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:51007)
0928 15:03:16.273909 (+ 20522us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.273926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.274305 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.274896 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.274901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.275487 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.275782 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.275795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.277493 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.277498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.279760 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.279771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.280940 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.280950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.281182 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.281833 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.281863 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.285726 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:16.290256 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.290260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.290263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.290855 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.291261 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.291266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.291269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.291446 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.292131 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:16.292135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:16.292263 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.293981 (+ 1718us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.294235 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20399,"mutex_wait_us":906,"thread_start_us":8460,"threads_started":1}
I20260928 15:03:16.296386 16404 heartbeater.cc:347] Connected to a master server at 127.9.12.252:45969
I20260928 15:03:16.296411 16406 heartbeater.cc:347] Connected to a master server at 127.9.12.253:46699
I20260928 15:03:16.296751 16404 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:16.296793 16406 heartbeater.cc:464] Registering TS with master...
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.194@KRBTEST.COM
I20260928 15:03:16.297741 16404 heartbeater.cc:511] Master 127.9.12.252:45969 requested a full tablet report, sending...
I20260928 15:03:16.297941 16405 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40395
I20260928 15:03:16.298058 16405 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:16.298543 16405 heartbeater.cc:511] Master 127.9.12.254:40395 requested a full tablet report, sending...
I20260928 15:03:16.298831 16406 heartbeater.cc:511] Master 127.9.12.253:46699 requested a full tablet report, sending...
I20260928 15:03:16.300454 16033 ts_manager.cc:194] Registered new tserver with Master: a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:16.300522 16224 ts_manager.cc:194] Registered new tserver with Master: a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:16.301497 15433 ts_manager.cc:194] Re-registered known tserver with Master: a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:16.302589 15433 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:51007
I20260928 15:03:16.305171 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.253908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.9.12.194:34177)
0928 15:03:16.267131 (+ 13223us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.267144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.278788 (+ 11644us) server_negotiation.cc:407] Connection header received
0928 15:03:16.278948 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.278957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.279312 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.279509 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.286516 (+ 7007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.289117 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.291077 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.292038 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.292149 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.298450 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.298460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.298471 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.298508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.302268 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.303061 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.303071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.303076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.303236 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.303614 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.303618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.303621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.303909 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.304224 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.304567 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.304823 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12981,"thread_start_us":12795,"threads_started":1}
I20260928 15:03:16.306388 15751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:16.389434 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15628
I20260928 15:03:16.405397 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:34177
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:40395
--webserver_port=35277
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:16.599853 16294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: NON_VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: true } } }
I20260928 15:03:16.600292 16294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:16.601619 16294 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: attempting to promote NON_VOTER 85712a8afd144249bf425b5e672ba007 to VOTER
I20260928 15:03:16.602161 16294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:16.603888 16043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Refusing update from remote peer 9c2e21c0253c4818bc507daf0f09d796: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 15:03:16.603890 16234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 LEARNER]: Refusing update from remote peer 9c2e21c0253c4818bc507daf0f09d796: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 15:03:16.604458 16418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 15:03:16.604918 16294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 15:03:16.607659 16417 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85712a8afd144249bf425b5e672ba007 (127.9.12.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.608099 16043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85712a8afd144249bf425b5e672ba007 (127.9.12.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.613420 16418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.613528 16294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.614096 16418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:16.614182 16294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:16.615113 16421 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.615514 16421 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:16.616676 16418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.616909 16418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:16.617304 16234 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85712a8afd144249bf425b5e672ba007 (127.9.12.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.620361 16262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c2e21c0253c4818bc507daf0f09d796" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:16.620659 16262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:03:16.737739 16416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:16.737977 16416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:16.738061 16416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:16.744712 16416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:16.744801 16416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:16.744905 16416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:16.754009 16416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:34177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=35277
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:16.755182 16416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:16.756510 16416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:16.769821 16426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.769793 16427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.769776 16429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:16.770721 16416 server_base.cc:1034] running on GCE node
I20260928 15:03:16.771267 16416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:16.772229 16416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:16.773489 16416 hybrid_clock.cc:648] HybridClock initialized: now 1790607796773458 us; error 31 us; skew 500 ppm
I20260928 15:03:16.773841 16416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:16.777261 16416 init.cc:377] Logged in from keytab as oryx/127.9.12.194@KRBTEST.COM (short username oryx)
I20260928 15:03:16.778748 16416 webserver.cc:460] Webserver started at http://127.9.12.194:35277/ using document root <none> and password file <none>
I20260928 15:03:16.779495 16416 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:16.779639 16416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:16.785601 16416 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 15:03:16.788355 16436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.789799 16416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:03:16.789922 16416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "89b2cbaffa5045f79052d8d1f710a76a"
format_stamp: "Formatted at 2026-09-28 15:03:13 on dist-test-slave-dmhs"
server_key: "312de9fcaccdd8a77c7e4d9005a4212a"
server_key_iv: "36349b2fd5220aebd8f85777d4a0457b"
server_key_version: "encryptionkey@0"
I20260928 15:03:16.790825 16416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:16.816210 16416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:16.820218 16416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:16.820500 16416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:16.821455 16416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:16.822697 16416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:16.822774 16416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.822845 16416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:16.822894 16416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.854686 16416 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:34177
I20260928 15:03:16.854736 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:34177 every 8 connection(s)
I20260928 15:03:16.856599 16416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:16.857077 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 16416
I20260928 15:03:16.857451 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15761
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.252@KRBTEST.COM
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.195@KRBTEST.COM
I20260928 15:03:16.898089 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.867380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:59503 (local address 127.9.12.252:45969)
0928 15:03:16.868919 (+ 1539us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.868927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.870188 (+ 1261us) server_negotiation.cc:407] Connection header received
0928 15:03:16.873293 (+ 3105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.873301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.873391 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.873473 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.876228 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.877771 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.881099 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.881979 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.882090 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.890059 (+ 7969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.890066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.890071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.890108 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.894115 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.895077 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.895084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.895089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.895250 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.895990 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.895995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.895998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.896357 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.896490 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.897451 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.897689 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":1183,"threads_started":1}
I20260928 15:03:16.898705 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.858469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.9.12.195:40671)
0928 15:03:16.868354 (+ 9885us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.868366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.870746 (+ 2380us) server_negotiation.cc:407] Connection header received
0928 15:03:16.870954 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.870966 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.871228 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.871372 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.873868 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.875896 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.878410 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.879445 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.879587 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.890182 (+ 10595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.890198 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.890215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.890256 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.895047 (+ 4791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.895901 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.895938 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.895942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.896098 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.896549 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.896554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.896557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.896956 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.897222 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.897786 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.898050 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9638,"thread_start_us":9402,"threads_started":1}
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
I20260928 15:03:16.900861 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.866643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.9.12.194:59503)
0928 15:03:16.869611 (+ 2968us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.869669 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.872938 (+ 3269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.874076 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.874089 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.874987 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.875962 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.875991 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.878098 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.878106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.880828 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.880846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.882509 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.882518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.882888 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.883706 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.883742 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.888432 (+ 4690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:16.894308 (+ 5876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.894316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.894332 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.894924 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.895596 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.895600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.895603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.895861 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.896594 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:16.896602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:16.897036 (+ 434us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.898347 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.898881 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2593,"thread_start_us":2337,"threads_started":1}
I20260928 15:03:16.901170 15884 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 28 15:03:16 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:16.902897 16549 heartbeater.cc:347] Connected to a master server at 127.9.12.252:45969
I20260928 15:03:16.903318 16549 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:16.904556 16549 heartbeater.cc:511] Master 127.9.12.252:45969 requested a full tablet report, sending...
I20260928 15:03:16.906601 16224 ts_manager.cc:194] Registered new tserver with Master: 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177)
I20260928 15:03:16.910940 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.890251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.9.12.194:35233)
0928 15:03:16.890998 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.891016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.891119 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.892224 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.892229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.893186 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.893617 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.893632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.895126 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.895131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.896845 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.896858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.897984 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.897991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.898172 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.898733 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.898758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.901750 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:16.908056 (+ 6306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.908061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.908065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.908552 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.909038 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.909045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.909051 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.909243 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.909978 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:16.909985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:16.910141 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.910278 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.910542 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":3,"thread_start_us":412,"threads_started":1}
I20260928 15:03:16.910954 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.891312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:35233 (local address 127.9.12.253:46699)
0928 15:03:16.891778 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.891784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.891803 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:03:16.891862 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.891869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.891943 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.892047 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.893801 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.894950 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.897026 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.897654 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.897762 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.901948 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.901954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.901957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.902053 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.907844 (+ 5791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.908675 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.908683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.908689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.908845 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.909374 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.909381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.909387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.909767 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.909919 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.910389 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.910612 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":409,"threads_started":1}
I20260928 15:03:16.911865 16555 heartbeater.cc:347] Connected to a master server at 127.9.12.253:46699
I20260928 15:03:16.911955 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.871711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:35329)
0928 15:03:16.889357 (+ 17646us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.889378 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.889526 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.892224 (+ 2698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.892229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.893149 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.893511 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.893529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.894967 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.894973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.896851 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.896864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.898107 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.898115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.898547 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.899412 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.899439 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.903786 (+ 4347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:16.908832 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.908840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.908846 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.909537 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.910058 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:16.910066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:16.910071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:16.910237 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:16.910968 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:16.910972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:16.911119 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 15:03:16.911236 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 15:03:16.911496 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17483,"mutex_wait_us":71,"thread_start_us":17239,"threads_started":1}
I20260928 15:03:16.912010 16555 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:16.912556 16555 heartbeater.cc:511] Master 127.9.12.253:46699 requested a full tablet report, sending...
I20260928 15:03:16.912953 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.872187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:35329 (local address 127.9.12.254:40395)
0928 15:03:16.888748 (+ 16561us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.888756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.889653 (+ 897us) server_negotiation.cc:407] Connection header received
0928 15:03:16.889725 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.889730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.891939 (+ 2209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.892052 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.893703 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.894801 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.897035 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.897708 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.897819 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.904055 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.904065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.904072 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.904157 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.908635 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.909695 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.909703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.909708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.909915 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.910380 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.910387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.910392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.910767 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.910912 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.912334 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.912572 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16391,"thread_start_us":16296,"threads_started":1}
I20260928 15:03:16.913712 16550 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40395
I20260928 15:03:16.913785 16033 ts_manager.cc:194] Registered new tserver with Master: 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177)
I20260928 15:03:16.913841 16550 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:16.914363 16550 heartbeater.cc:511] Master 127.9.12.254:40395 requested a full tablet report, sending...
I20260928 15:03:16.915354 15433 ts_manager.cc:194] Re-registered known tserver with Master: 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177)
I20260928 15:03:16.916419 15433 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:35329
I20260928 15:03:16.993840 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15761
I20260928 15:03:17.009873 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:40671
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:40395
--webserver_port=36281
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:17.304738 16405 heartbeater.cc:503] Master 127.9.12.254:40395 was elected leader, sending a full tablet report...
W20260928 15:03:17.328956 16562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:17.329217 16562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:17.329393 16562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:17.336356 16562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:17.336468 16562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:17.336621 16562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:17.346992 16562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:40671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=36281
--tserver_master_addrs=127.9.12.254:40395,127.9.12.253:46699,127.9.12.252:45969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:17.348500 16562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:17.349768 16562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:17.363020 16568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:17.362991 16570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:17.363063 16567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:17.365548 16562 server_base.cc:1034] running on GCE node
I20260928 15:03:17.366170 16562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:17.367066 16562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:17.368328 16562 hybrid_clock.cc:648] HybridClock initialized: now 1790607797368243 us; error 58 us; skew 500 ppm
I20260928 15:03:17.368654 16562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:17 dist-test-slave-dmhs krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:17.371476 16562 init.cc:377] Logged in from keytab as oryx/127.9.12.195@KRBTEST.COM (short username oryx)
I20260928 15:03:17.372879 16562 webserver.cc:460] Webserver started at http://127.9.12.195:36281/ using document root <none> and password file <none>
I20260928 15:03:17.373615 16562 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:17.373785 16562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:17.379487 16562 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:03:17.382278 16577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:17.383697 16562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:17.383865 16562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "276bf8b78da443fe9cbbdf6c6c172769"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-dmhs"
server_key: "616d7142a7026111016f067e767b8a85"
server_key_iv: "c3118fdb4245d9a58d2e5c852d8f2892"
server_key_version: "encryptionkey@0"
I20260928 15:03:17.385001 16562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:17.402244 16562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:17.405944 16562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:17.406211 16562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:17.407271 16562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:17.408645 16562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:17.408767 16562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:17.408885 16562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:17.408969 16562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:17.442315 16562 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:40671
I20260928 15:03:17.442399 16689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:40671 every 8 connection(s)
I20260928 15:03:17.443837 16562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:17.449755 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 16562
Sep 28 15:03:17 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.253@KRBTEST.COM
Sep 28 15:03:17 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.252@KRBTEST.COM
Sep 28 15:03:17 dist-test-slave-dmhs krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:17.489097 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.446770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:47435 (local address 127.9.12.252:45969)
0928 15:03:17.459333 (+ 12563us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.459341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.459367 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:03:17.459442 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.459449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.459548 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.459683 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.462836 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.463976 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.466494 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.467097 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.468002 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.472484 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.472493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.472501 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.472549 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.477767 (+ 5218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.485637 (+ 7870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.485642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.485645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.485824 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.486310 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.486316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.486318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.486653 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.486799 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.488464 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.488706 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12412,"thread_start_us":13022,"threads_started":1}
I20260928 15:03:17.490262 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.446845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:37333 (local address 127.9.12.253:46699)
0928 15:03:17.462013 (+ 15168us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.462019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.462038 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:03:17.462107 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.462112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.462192 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.462297 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.463976 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.465059 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.466970 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.467570 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.468247 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.471889 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.471895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.471899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.471946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.476218 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.485714 (+ 9496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.485719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.485724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.485903 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.486357 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.486362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.486367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.486653 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.486815 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.489669 (+ 2854us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.489900 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15050,"thread_start_us":15520,"threads_started":1}
I20260928 15:03:17.491267 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.448818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.9.12.195:47435)
0928 15:03:17.452689 (+ 3871us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.452711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.456813 (+ 4102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.460018 (+ 3205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.460034 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.461753 (+ 1719us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:17.462656 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.462676 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.464156 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.464161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.466304 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.466323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.467322 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.467328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.467591 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.468433 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:17.468459 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:17.472233 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:17.477888 (+ 5655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:17.477894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:17.477900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:17.485526 (+ 7626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:17.485952 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:17.485958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:17.485961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:17.486168 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:17.486900 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:17.486907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:17.487394 (+ 487us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.487739 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.489853 (+ 2114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3717,"thread_start_us":620,"threads_started":1}
I20260928 15:03:17.492111 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.447503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.9.12.195:37333)
0928 15:03:17.449661 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.449706 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.456403 (+ 6697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.462876 (+ 6473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.462882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.463320 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:17.463531 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.463541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.465221 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.465227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.466778 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.466791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.467773 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.467781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.467982 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.468626 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:17.468647 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:17.471691 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:17.476474 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:17.476483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:17.476500 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:17.484933 (+ 8433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:17.486014 (+ 1081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:17.486020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:17.486025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:17.486245 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:17.486898 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:17.486907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:17.489424 (+ 2517us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.489554 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.491704 (+ 2150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"mutex_wait_us":4160,"spinlock_wait_cycles":20352,"thread_start_us":13112,"threads_started":1}
I20260928 15:03:17.493253 16692 heartbeater.cc:347] Connected to a master server at 127.9.12.253:46699
I20260928 15:03:17.493283 16690 heartbeater.cc:347] Connected to a master server at 127.9.12.252:45969
I20260928 15:03:17.493564 16690 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:17.493568 16692 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:17.494513 16690 heartbeater.cc:511] Master 127.9.12.252:45969 requested a full tablet report, sending...
I20260928 15:03:17.494643 16692 heartbeater.cc:511] Master 127.9.12.253:46699 requested a full tablet report, sending...
I20260928 15:03:17.496369 16033 ts_manager.cc:194] Registered new tserver with Master: 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195:40671)
I20260928 15:03:17.496430 16224 ts_manager.cc:194] Registered new tserver with Master: 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195:40671)
I20260928 15:03:17.498111 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.449233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:41793)
0928 15:03:17.461038 (+ 11805us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.461061 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.461201 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.461873 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.461886 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.462617 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:17.463107 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.463124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.464814 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.464823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.467016 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.467035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.469184 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.469192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.469486 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.472505 (+ 3019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:17.472530 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:17.490090 (+ 17560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:17.494232 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:17.494238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:17.494247 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:17.495424 (+ 1177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:17.495890 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:17.495895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:17.495899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:17.496050 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:17.496820 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:17.496825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:17.497000 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.497193 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.497485 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11679,"mutex_wait_us":6,"thread_start_us":3159,"threads_started":1}
I20260928 15:03:17.499441 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.448240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:41793 (local address 127.9.12.254:40395)
0928 15:03:17.460589 (+ 12349us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.460598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.461331 (+ 733us) server_negotiation.cc:407] Connection header received
0928 15:03:17.461418 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.461426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.461549 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.461663 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.463324 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.464599 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.467225 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.468717 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.468856 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.490354 (+ 21498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.490364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.490371 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.490418 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.494087 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.495584 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.495589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.495593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.495750 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.496207 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.496212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.496216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.496680 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.498445 (+ 1765us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.498613 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.498889 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12190,"thread_start_us":12069,"threads_started":1}
I20260928 15:03:17.500388 16691 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40395
I20260928 15:03:17.500586 16691 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:17.501273 16691 heartbeater.cc:511] Master 127.9.12.254:40395 requested a full tablet report, sending...
I20260928 15:03:17.502219 15433 ts_manager.cc:194] Re-registered known tserver with Master: 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195:40671)
I20260928 15:03:17.503221 15433 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:41793
I20260928 15:03:17.508972 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.474594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.9.12.252:45969)
0928 15:03:17.482763 (+ 8169us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.482770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.482813 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:03:17.482867 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.482875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.482956 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.483083 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.485145 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.493824 (+ 8679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.495869 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.496615 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.496764 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.500775 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.500781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.500785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.500828 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.504634 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.505894 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.506144 (+ 250us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.506152 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.506339 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.506901 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.506906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.506909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.507331 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.507621 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.508464 (+ 843us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.508692 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8038,"thread_start_us":7903,"threads_started":1}
I20260928 15:03:17.514339 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.472989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.9.12.253:46699)
0928 15:03:17.490485 (+ 17496us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.490492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.490513 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:03:17.490555 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.490576 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.490649 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.490739 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.501023 (+ 10284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.501763 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.503735 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.504488 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.504620 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.508004 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.508009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.508012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.508041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.511617 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.512213 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.512218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.512221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.512395 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.512767 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.512770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.512772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.513104 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.513358 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.513774 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.513999 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17391,"thread_start_us":17699,"threads_started":1}
I20260928 15:03:17.515595 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.474208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.9.12.254:40395)
0928 15:03:17.483654 (+ 9446us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.483660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.488975 (+ 5315us) server_negotiation.cc:407] Connection header received
0928 15:03:17.489032 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.489037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.489138 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.489464 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.500218 (+ 10754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.502786 (+ 2568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.504452 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.505072 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.505196 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.508879 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.508889 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.508896 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.508935 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.512255 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.513002 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.513007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.513011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.513179 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.513552 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.513556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.513558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.513885 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.514130 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.515183 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.515342 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9305,"thread_start_us":9189,"threads_started":1}
I20260928 15:03:17.519483 15433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:17.527998 15433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:17.567936 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.546715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:40671 (local address 127.0.0.1:49132)
0928 15:03:17.558505 (+ 11790us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.558521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.558680 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.559435 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.559438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.559456 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:17.559715 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.559721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.561729 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.561736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.563703 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.563721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.566786 (+ 3065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.566792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.567057 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.567099 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.567193 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.567357 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11694,"thread_start_us":366,"threads_started":1}
I20260928 15:03:17.568032 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.547195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.9.12.195:40671)
0928 15:03:17.554297 (+ 7102us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.554309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.558633 (+ 4324us) server_negotiation.cc:407] Connection header received
0928 15:03:17.558841 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.558853 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.559215 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.559457 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:17.559898 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.561588 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.563965 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.566228 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.566350 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.566638 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.567266 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.567497 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6879,"thread_start_us":6640,"threads_started":1}
I20260928 15:03:17.569258 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.547894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37065 (local address 127.0.0.1:60832)
0928 15:03:17.559950 (+ 12056us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.559972 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.560067 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.560919 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.560923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.560948 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:17.561298 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.561308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.563169 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.563175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.565782 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.565797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.568689 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.568693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.568813 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.568831 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.568897 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.568977 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11956,"thread_start_us":11800,"threads_started":1}
I20260928 15:03:17.569612 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.548320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.9.12.193:37065)
0928 15:03:17.557274 (+ 8954us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.557284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.560074 (+ 2790us) server_negotiation.cc:407] Connection header received
0928 15:03:17.560209 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.560220 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.560526 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.560721 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:17.561556 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.563006 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.566008 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.568038 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.568153 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.568377 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.568983 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.569184 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8783,"thread_start_us":9649,"threads_started":1}
I20260928 15:03:17.570366 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.547909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.9.12.194:34177)
0928 15:03:17.560706 (+ 12797us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.560719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.560740 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:03:17.560810 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.560814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.561057 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.561232 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:17.562472 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.563727 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.565826 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.567476 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.567606 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.567865 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.569790 (+ 1925us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.570025 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12636,"thread_start_us":12525,"threads_started":1}
I20260928 15:03:17.571542 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.547322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:34177 (local address 127.0.0.1:55558)
0928 15:03:17.559075 (+ 11753us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.559088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.559165 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.561809 (+ 2644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.561812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.561835 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:17.562279 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.562293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.563853 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.563860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.565659 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.565671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.568128 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.568137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.569429 (+ 1292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.569459 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.569544 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.571110 (+ 1566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11658,"thread_start_us":10964,"threads_started":1}
I20260928 15:03:17.572296 16625 tablet_service.cc:1467] Processing CreateTablet for tablet ff044953c0204785b72bf9e263c5ba33 (DEFAULT_TABLE table=test-table [id=8f8c91f4db6446c2981637d71a5d1abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:17.573588 16625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff044953c0204785b72bf9e263c5ba33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:17.573539 16339 tablet_service.cc:1467] Processing CreateTablet for tablet ff044953c0204785b72bf9e263c5ba33 (DEFAULT_TABLE table=test-table [id=8f8c91f4db6446c2981637d71a5d1abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:17.575114 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff044953c0204785b72bf9e263c5ba33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:17.575214 16484 tablet_service.cc:1467] Processing CreateTablet for tablet ff044953c0204785b72bf9e263c5ba33 (DEFAULT_TABLE table=test-table [id=8f8c91f4db6446c2981637d71a5d1abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:17.576802 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff044953c0204785b72bf9e263c5ba33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:17.587106 16725 tablet_bootstrap.cc:492] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Bootstrap starting.
I20260928 15:03:17.591250 16726 tablet_bootstrap.cc:492] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Bootstrap starting.
I20260928 15:03:17.591312 16725 tablet_bootstrap.cc:654] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:17.592801 16727 tablet_bootstrap.cc:492] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Bootstrap starting.
I20260928 15:03:17.592888 16725 log.cc:824] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:17.595091 16726 tablet_bootstrap.cc:654] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:17.596472 16727 tablet_bootstrap.cc:654] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:17.596839 16726 log.cc:824] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:17.597620 16727 log.cc:824] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:17.602878 16725 tablet_bootstrap.cc:492] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: No bootstrap required, opened a new log
I20260928 15:03:17.602962 16727 tablet_bootstrap.cc:492] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: No bootstrap required, opened a new log
I20260928 15:03:17.603173 16725 ts_tablet_manager.cc:1397] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260928 15:03:17.603286 16727 ts_tablet_manager.cc:1397] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260928 15:03:17.603439 16726 tablet_bootstrap.cc:492] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: No bootstrap required, opened a new log
I20260928 15:03:17.603755 16726 ts_tablet_manager.cc:1397] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260928 15:03:17.611562 16727 raft_consensus.cc:348] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.611982 16727 raft_consensus.cc:374] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:17.612104 16727 raft_consensus.cc:729] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b2cbaffa5045f79052d8d1f710a76a, State: Initialized, Role: FOLLOWER
I20260928 15:03:17.612855 16727 consensus_queue.cc:260] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.614454 16725 raft_consensus.cc:348] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.614899 16727 ts_tablet_manager.cc:1428] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 15:03:17.614930 16725 raft_consensus.cc:374] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:17.615083 16725 raft_consensus.cc:729] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276bf8b78da443fe9cbbdf6c6c172769, State: Initialized, Role: FOLLOWER
I20260928 15:03:17.615615 16550 heartbeater.cc:503] Master 127.9.12.254:40395 was elected leader, sending a full tablet report...
I20260928 15:03:17.615792 16725 consensus_queue.cc:260] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.616406 16726 raft_consensus.cc:348] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.616884 16726 raft_consensus.cc:374] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:17.617004 16726 raft_consensus.cc:729] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a591c02989be4f2c96c8bb60ee820a66, State: Initialized, Role: FOLLOWER
I20260928 15:03:17.617728 16726 consensus_queue.cc:260] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.622365 16726 ts_tablet_manager.cc:1428] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20260928 15:03:17.628430 16691 heartbeater.cc:503] Master 127.9.12.254:40395 was elected leader, sending a full tablet report...
I20260928 15:03:17.628796 16725 ts_tablet_manager.cc:1428] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Time spent starting tablet: real 0.026s user 0.016s sys 0.010s
W20260928 15:03:17.642025 16558 tablet.cc:2369] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:17.706116 16693 tablet.cc:2369] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:17.762143 16407 tablet.cc:2369] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:17.769439 16732 raft_consensus.cc:482] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:17.769773 16732 raft_consensus.cc:504] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.771492 16732 leader_election.cc:288] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177), a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:17.777727 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.772094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:52201 (local address 127.9.12.193:37065)
0928 15:03:17.772370 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.772376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.772396 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:03:17.772601 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.772605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.772664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.772757 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:17.773261 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.773976 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.775793 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.776718 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.776895 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.776970 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.777332 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.777456 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 15:03:17.777774 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.771852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37065 (local address 127.9.12.195:52201)
0928 15:03:17.772228 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.772250 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.772432 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.772773 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.772777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.772805 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:17.773122 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.773132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.774147 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.774150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.775657 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.775669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.776917 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.776923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.777147 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.777189 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.777307 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.777443 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":2}
I20260928 15:03:17.777819 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.771852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:33849 (local address 127.9.12.194:34177)
0928 15:03:17.772120 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.772128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.772403 (+ 275us) server_negotiation.cc:407] Connection header received
0928 15:03:17.772603 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.772607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.772667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.772762 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:17.773232 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.773981 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.775804 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.776864 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.777034 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.777105 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.777425 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.777549 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:03:17.777902 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.771946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:34177 (local address 127.9.12.195:33849)
0928 15:03:17.772300 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.772320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.772452 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.772823 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.772828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.772852 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:17.773086 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.773098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.774113 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.774118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.775656 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.775668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.777028 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.777034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.777274 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.777297 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:03:17.777398 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:03:17.777530 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2}
I20260928 15:03:17.778916 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff044953c0204785b72bf9e263c5ba33" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a591c02989be4f2c96c8bb60ee820a66" is_pre_election: true
I20260928 15:03:17.779062 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff044953c0204785b72bf9e263c5ba33" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b2cbaffa5045f79052d8d1f710a76a" is_pre_election: true
I20260928 15:03:17.779551 16359 raft_consensus.cc:2454] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 0.
I20260928 15:03:17.779572 16504 raft_consensus.cc:2454] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 0.
I20260928 15:03:17.780369 16580 leader_election.cc:302] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 276bf8b78da443fe9cbbdf6c6c172769, 89b2cbaffa5045f79052d8d1f710a76a, a591c02989be4f2c96c8bb60ee820a66; no voters:
I20260928 15:03:17.780813 16732 raft_consensus.cc:2781] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:17.780966 16732 raft_consensus.cc:482] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:17.781072 16732 raft_consensus.cc:3037] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:17.783403 16732 raft_consensus.cc:504] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.784054 16732 leader_election.cc:288] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 election: Requested vote from peers 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177), a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:17.784528 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff044953c0204785b72bf9e263c5ba33" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b2cbaffa5045f79052d8d1f710a76a"
I20260928 15:03:17.784622 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff044953c0204785b72bf9e263c5ba33" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a591c02989be4f2c96c8bb60ee820a66"
I20260928 15:03:17.784772 16504 raft_consensus.cc:3037] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:17.784878 16359 raft_consensus.cc:3037] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:17.787168 16359 raft_consensus.cc:2454] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 1.
I20260928 15:03:17.787570 16580 leader_election.cc:302] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 276bf8b78da443fe9cbbdf6c6c172769, a591c02989be4f2c96c8bb60ee820a66; no voters:
I20260928 15:03:17.787642 16504 raft_consensus.cc:2454] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 1.
I20260928 15:03:17.787868 16732 raft_consensus.cc:2781] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:17.788414 16732 raft_consensus.cc:686] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 LEADER]: Becoming Leader. State: Replica: 276bf8b78da443fe9cbbdf6c6c172769, State: Running, Role: LEADER
I20260928 15:03:17.789361 16732 consensus_queue.cc:237] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:17.795555 15433 catalog_manager.cc:5495] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "276bf8b78da443fe9cbbdf6c6c172769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:17.865262 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.859607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.9.12.195:40671)
0928 15:03:17.859829 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.859835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.859854 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:03:17.860096 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.860106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.860188 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.860379 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:17.860771 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.861713 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.863036 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.863448 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.863581 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.864096 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:17.864677 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:17.864829 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.864910 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.864990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:03:17.874274 16504 raft_consensus.cc:1261] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Refusing update from remote peer 276bf8b78da443fe9cbbdf6c6c172769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:17.874285 16359 raft_consensus.cc:1261] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Refusing update from remote peer 276bf8b78da443fe9cbbdf6c6c172769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:17.875290 16732 consensus_queue.cc:1035] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:17.875610 16734 consensus_queue.cc:1035] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:17.888978 16739 mvcc.cc:205] Tried to move back new op lower bound from 7334329540083798016 to 7334329539749310464. Current Snapshot: MvccSnapshot[applied={T|T < 7334329540083798016}]
I20260928 15:03:17.889699 16737 mvcc.cc:205] Tried to move back new op lower bound from 7334329540083798016 to 7334329539749310464. Current Snapshot: MvccSnapshot[applied={T|T < 7334329540083798016}]
I20260928 15:03:17.912694 15432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:17.913075 15432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:17.919487 15432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2e21c0253c4818bc507daf0f09d796: Sending DeleteTablet for 3 replicas of tablet ff044953c0204785b72bf9e263c5ba33
I20260928 15:03:17.920877 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet ff044953c0204785b72bf9e263c5ba33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:17 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:60832
I20260928 15:03:17.921005 16484 tablet_service.cc:1514] Processing DeleteTablet for tablet ff044953c0204785b72bf9e263c5ba33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:17 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:55558
I20260928 15:03:17.921268 16625 tablet_service.cc:1514] Processing DeleteTablet for tablet ff044953c0204785b72bf9e263c5ba33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:17 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:49132
I20260928 15:03:17.921918 16747 tablet_replica.cc:330] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: stopping tablet replica
I20260928 15:03:17.922379 16747 raft_consensus.cc:2229] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:17.923092 16747 raft_consensus.cc:2258] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:17.932315 16747 ts_tablet_manager.cc:1905] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:17.934156 16748 tablet_replica.cc:330] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: stopping tablet replica
I20260928 15:03:17.934739 16748 raft_consensus.cc:2229] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:17.935314 16748 raft_consensus.cc:2258] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:17.936923 16749 tablet_replica.cc:330] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: stopping tablet replica
I20260928 15:03:17.937335 16749 raft_consensus.cc:2229] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:17.938083 16749 raft_consensus.cc:2258] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:17.940078 16749 ts_tablet_manager.cc:1905] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:17.940747 16748 ts_tablet_manager.cc:1905] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:17.941511 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15401
I20260928 15:03:17.942137 16747 ts_tablet_manager.cc:1918] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:17.942497 16747 log.cc:1196] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/ff044953c0204785b72bf9e263c5ba33
I20260928 15:03:17.942521 15463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:17.943125 16747 ts_tablet_manager.cc:1939] T ff044953c0204785b72bf9e263c5ba33 P a591c02989be4f2c96c8bb60ee820a66: Deleting consensus metadata
I20260928 15:03:17.946769 16749 ts_tablet_manager.cc:1918] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:17.946977 16749 log.cc:1196] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/ff044953c0204785b72bf9e263c5ba33
I20260928 15:03:17.947647 16749 ts_tablet_manager.cc:1939] T ff044953c0204785b72bf9e263c5ba33 P 276bf8b78da443fe9cbbdf6c6c172769: Deleting consensus metadata
I20260928 15:03:17.950898 16748 ts_tablet_manager.cc:1918] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:17.951071 16748 log.cc:1196] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/ff044953c0204785b72bf9e263c5ba33
I20260928 15:03:17.951841 16748 ts_tablet_manager.cc:1939] T ff044953c0204785b72bf9e263c5ba33 P 89b2cbaffa5045f79052d8d1f710a76a: Deleting consensus metadata
I20260928 15:03:17.968628 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.924325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.9.12.253:46699)
0928 15:03:17.924541 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.924549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.936043 (+ 11494us) server_negotiation.cc:407] Connection header received
0928 15:03:17.936294 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.936304 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.936423 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.936548 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.939382 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.940601 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.949685 (+ 9084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.950656 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.950788 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.960093 (+ 9305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.960100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.960104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.960153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.964439 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.965463 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.965470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.965474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.965720 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.966193 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.966198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.966202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.966602 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.966839 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.967554 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.968037 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:03:17.984580 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.940968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.9.12.252:45969)
0928 15:03:17.941179 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.941183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.968138 (+ 26955us) server_negotiation.cc:407] Connection header received
0928 15:03:17.968439 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.968444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.968534 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.968617 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.970663 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.971841 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.973702 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.974335 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.974471 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.977849 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:17.977856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:17.977861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:17.977902 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:17.981716 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.982472 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.982477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.982481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.982637 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:17.983056 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:17.983061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:17.983065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:17.983413 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:17.983661 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:17.984005 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 15:03:17.984224 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:03:18.082585 15420 catalog_manager.cc:4841] TS 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177): tablet ff044953c0204785b72bf9e263c5ba33 (table test-table [id=8f8c91f4db6446c2981637d71a5d1abf]) successfully deleted
I20260928 15:03:18.082553 15418 catalog_manager.cc:4841] TS 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195:40671): tablet ff044953c0204785b72bf9e263c5ba33 (table test-table [id=8f8c91f4db6446c2981637d71a5d1abf]) successfully deleted
W20260928 15:03:18.083073 15418 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.12.195:40671, user_credentials={real_user=slave}} blocked reactor thread for 133647us
I20260928 15:03:18.084441 15419 catalog_manager.cc:4841] TS a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065): tablet ff044953c0204785b72bf9e263c5ba33 (table test-table [id=8f8c91f4db6446c2981637d71a5d1abf]) successfully deleted
W20260928 15:03:18.084681 15419 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.12.193:37065, user_credentials={real_user=slave}} blocked reactor thread for 139735us
I20260928 15:03:18.089548 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15401
I20260928 15:03:18.142782 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.122611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.9.12.252:45969)
0928 15:03:18.122794 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:18.122800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:18.128711 (+ 5911us) server_negotiation.cc:407] Connection header received
0928 15:03:18.128996 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:18.129002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:18.129080 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:18.129178 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:18.131159 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.131894 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.133664 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.134261 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.134347 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.137551 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:18.137555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:18.137565 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:18.137598 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:18.140199 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:18.140835 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:18.140845 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:18.140847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:18.140960 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:18.141320 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:18.141324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:18.141327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:18.141687 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:18.141903 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:18.142240 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:03:18.142397 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:03:18.144670 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.122513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.9.12.253:46699)
0928 15:03:18.122728 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:18.122734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:18.127145 (+ 4411us) server_negotiation.cc:407] Connection header received
0928 15:03:18.127358 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:18.127363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:18.127446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:18.127551 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:18.130673 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.131741 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.133117 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.133520 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.133650 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.137238 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:18.137244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:18.137249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:18.137288 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:18.140772 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:18.141547 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:18.141552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:18.141555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:18.141700 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:18.142923 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:18.142929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:18.142932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:18.143317 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:18.143542 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:18.144102 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 15:03:18.144313 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:03:18.895884 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.895172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:37241)
0928 15:03:18.895615 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:18.895728 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
W20260928 15:03:18.896355 16405 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:40395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:18.903087 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.902204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:33963)
0928 15:03:18.902763 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:18.902891 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":348,"threads_started":1}
I20260928 15:03:18.903122 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.901291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:38783)
0928 15:03:18.902753 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:18.902892 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
W20260928 15:03:18.903462 16550 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:40395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
W20260928 15:03:18.903496 16691 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:40395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.420194 16762 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c2e21c0253c4818bc507daf0f09d796)
I20260928 15:03:19.420538 16762 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:19.420642 16763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c2e21c0253c4818bc507daf0f09d796)
I20260928 15:03:19.420904 16763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:19.421855 16762 leader_election.cc:288] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395), 85712a8afd144249bf425b5e672ba007 (127.9.12.252:45969)
I20260928 15:03:19.423354 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.422017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36164)
0928 15:03:19.422732 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.422993 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":404,"threads_started":1}
W20260928 15:03:19.423782 16019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.432651 16763 leader_election.cc:288] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395), e4144ee7a7954d74a6ecef86288072df (127.9.12.253:46699)
I20260928 15:03:19.434480 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.422305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36176)
0928 15:03:19.434036 (+ 11731us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.434202 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11602,"mutex_wait_us":24,"thread_start_us":11402,"threads_started":1}
W20260928 15:03:19.434935 16210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.436007 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.432999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36186)
0928 15:03:19.434329 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.434577 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1187}
W20260928 15:03:19.436239 16019 leader_election.cc:334] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.437636 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.437248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36188)
0928 15:03:19.437401 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.437481 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:03:19.437906 16210 leader_election.cc:334] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.442427 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.422192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:45969 (local address 127.0.0.1:50788)
0928 15:03:19.429219 (+ 7027us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.429241 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:19.429390 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:19.435575 (+ 6185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:19.435580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:19.435610 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:19.435972 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.436008 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.437956 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.437963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.439972 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.439988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.441636 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.441641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.441839 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.441872 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:03:19.441966 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:03:19.442095 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6903,"spinlock_wait_cycles":1280,"thread_start_us":11858,"threads_started":1}
I20260928 15:03:19.442533 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.422297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46699 (local address 127.0.0.1:50788)
0928 15:03:19.433320 (+ 11023us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.433343 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:19.433502 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:19.435998 (+ 2496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:19.436005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:19.436073 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:19.436564 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.436577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.438020 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.438025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.440005 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.440018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.441638 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.441643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.441838 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.441872 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:03:19.441955 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:03:19.442158 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10867,"thread_start_us":10703,"threads_started":1}
I20260928 15:03:19.442672 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.422604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.9.12.252:45969)
0928 15:03:19.435253 (+ 12649us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:19.435259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:19.435277 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:19.435343 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:19.435347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:19.435410 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:19.435554 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:19.436613 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.437815 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.440166 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.441440 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.441570 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.441718 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:19.442005 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 15:03:19.442140 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12510,"thread_start_us":12361,"threads_started":1}
I20260928 15:03:19.443239 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.434288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.9.12.253:46699)
0928 15:03:19.435492 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:19.435501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:19.435532 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:03:19.435610 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:19.435633 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:19.435735 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:19.436639 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:19.436891 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.437828 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.440191 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.441455 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.441570 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.442719 (+ 1149us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:19.442773 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 15:03:19.442902 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":889,"threads_started":1}
I20260928 15:03:19.443519 16234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4144ee7a7954d74a6ecef86288072df" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85712a8afd144249bf425b5e672ba007" is_pre_election: true
I20260928 15:03:19.443755 16043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85712a8afd144249bf425b5e672ba007" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4144ee7a7954d74a6ecef86288072df" is_pre_election: true
I20260928 15:03:19.443867 16234 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4144ee7a7954d74a6ecef86288072df in term 1.
I20260928 15:03:19.443998 16043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85712a8afd144249bf425b5e672ba007 in term 1.
I20260928 15:03:19.444480 16017 leader_election.cc:302] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85712a8afd144249bf425b5e672ba007, e4144ee7a7954d74a6ecef86288072df; no voters: 9c2e21c0253c4818bc507daf0f09d796
I20260928 15:03:19.444458 16209 leader_election.cc:302] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85712a8afd144249bf425b5e672ba007, e4144ee7a7954d74a6ecef86288072df; no voters: 9c2e21c0253c4818bc507daf0f09d796
I20260928 15:03:19.444823 16762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:03:19.444937 16762 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c2e21c0253c4818bc507daf0f09d796)
I20260928 15:03:19.445040 16762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:03:19.445076 16763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:03:19.445194 16763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c2e21c0253c4818bc507daf0f09d796)
I20260928 15:03:19.445281 16763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:03:19.448239 16762 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:19.448235 16763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:19.448966 16762 leader_election.cc:288] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [CANDIDATE]: Term 2 election: Requested vote from peers 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395), 85712a8afd144249bf425b5e672ba007 (127.9.12.252:45969)
I20260928 15:03:19.449175 16763 leader_election.cc:288] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 2 election: Requested vote from peers 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395), e4144ee7a7954d74a6ecef86288072df (127.9.12.253:46699)
I20260928 15:03:19.449436 16234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4144ee7a7954d74a6ecef86288072df" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85712a8afd144249bf425b5e672ba007"
I20260928 15:03:19.449584 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.449112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36196)
0928 15:03:19.449344 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.449449 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260928 15:03:19.449719 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.449326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36208)
0928 15:03:19.449459 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.449540 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 15:03:19.449736 16234 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e4144ee7a7954d74a6ecef86288072df in current term 2: Already voted for candidate 85712a8afd144249bf425b5e672ba007 in this term.
I20260928 15:03:19.450027 16043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85712a8afd144249bf425b5e672ba007" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4144ee7a7954d74a6ecef86288072df"
I20260928 15:03:19.450286 16043 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85712a8afd144249bf425b5e672ba007 in current term 2: Already voted for candidate e4144ee7a7954d74a6ecef86288072df in this term.
I20260928 15:03:19.450664 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.450256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36222)
0928 15:03:19.450438 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.450495 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:03:19.450969 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.450541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36234)
0928 15:03:19.450725 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.450823 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260928 15:03:19.450944 16210 leader_election.cc:334] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.451104 16210 leader_election.cc:302] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85712a8afd144249bf425b5e672ba007; no voters: 9c2e21c0253c4818bc507daf0f09d796, e4144ee7a7954d74a6ecef86288072df
W20260928 15:03:19.451188 16019 leader_election.cc:334] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:19.451318 16019 leader_election.cc:302] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4144ee7a7954d74a6ecef86288072df; no voters: 85712a8afd144249bf425b5e672ba007, 9c2e21c0253c4818bc507daf0f09d796
I20260928 15:03:19.451395 16763 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 15:03:19.451637 16762 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 15:03:19.897681 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.897022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:34245)
0928 15:03:19.897439 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.897519 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260928 15:03:19.905086 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.904047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:36837)
0928 15:03:19.904760 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.904859 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":535,"threads_started":1}
I20260928 15:03:19.905063 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.904055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:55517)
0928 15:03:19.904736 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.904853 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":461,"threads_started":1}
I20260928 15:03:20.899296 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:20.898539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:49205)
0928 15:03:20.899044 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:20.899125 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260928 15:03:20.907246 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:20.906162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:53727)
0928 15:03:20.906922 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:20.907052 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":485,"threads_started":1}
I20260928 15:03:20.907498 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:20.906171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:49141)
0928 15:03:20.907160 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:20.907269 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":783,"threads_started":1}
I20260928 15:03:21.315169 16778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:21.315449 16778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:21.316156 16778 leader_election.cc:288] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395), e4144ee7a7954d74a6ecef86288072df (127.9.12.253:46699)
I20260928 15:03:21.316834 16043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85712a8afd144249bf425b5e672ba007" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4144ee7a7954d74a6ecef86288072df" is_pre_election: true
I20260928 15:03:21.317077 16043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85712a8afd144249bf425b5e672ba007 in term 2.
I20260928 15:03:21.317237 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.316387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36498)
0928 15:03:21.316870 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.316969 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":247,"threads_started":1}
I20260928 15:03:21.317575 16209 leader_election.cc:302] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85712a8afd144249bf425b5e672ba007, e4144ee7a7954d74a6ecef86288072df; no voters:
I20260928 15:03:21.317880 16778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 15:03:21.318022 16778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:21.318149 16778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:03:21.318938 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.318579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36508)
0928 15:03:21.318709 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.318780 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:03:21.319121 16210 leader_election.cc:334] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:21.321090 16778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:21.321823 16778 leader_election.cc:288] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 3 election: Requested vote from peers 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395), e4144ee7a7954d74a6ecef86288072df (127.9.12.253:46699)
I20260928 15:03:21.322285 16043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85712a8afd144249bf425b5e672ba007" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4144ee7a7954d74a6ecef86288072df"
I20260928 15:03:21.322379 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.321850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36520)
0928 15:03:21.322083 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.322186 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 15:03:21.322497 16043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:03:21.323239 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.322926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36524)
0928 15:03:21.323075 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.323144 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:03:21.323448 16210 leader_election.cc:334] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
I20260928 15:03:21.324774 16043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85712a8afd144249bf425b5e672ba007 in term 3.
I20260928 15:03:21.325136 16209 leader_election.cc:302] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85712a8afd144249bf425b5e672ba007, e4144ee7a7954d74a6ecef86288072df; no voters: 9c2e21c0253c4818bc507daf0f09d796
I20260928 15:03:21.325414 16778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:03:21.325780 16778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [term 3 LEADER]: Becoming Leader. State: Replica: 85712a8afd144249bf425b5e672ba007, State: Running, Role: LEADER
I20260928 15:03:21.326159 16778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } }
I20260928 15:03:21.327827 16781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85712a8afd144249bf425b5e672ba007. Latest consensus state: current_term: 3 leader_uuid: "85712a8afd144249bf425b5e672ba007" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:21.328073 16781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:21.328712 16783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:21.332162 16783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:21.333099 16783 catalog_manager.cc:1259] Loaded cluster ID: 00cddc8856314a5f89a116473f4c4dff
I20260928 15:03:21.333194 16783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:21.334313 16783 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:21.335111 16783 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Loaded TSK: 0
I20260928 15:03:21.335958 16783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:21.408041 16223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:21.414788 16223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:21.418756 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.418257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36538)
0928 15:03:21.418460 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.418533 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:03:21.418781 16043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [term 3 FOLLOWER]: Refusing update from remote peer 85712a8afd144249bf425b5e672ba007: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 15:03:21.419494 16781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 15:03:21.419654 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.419329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36550)
0928 15:03:21.419489 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.419558 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:03:21.419936 16210 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 -> Peer 9c2e21c0253c4818bc507daf0f09d796 (127.9.12.254:40395): Couldn't send request to peer 9c2e21c0253c4818bc507daf0f09d796. Status: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 15:03:21.421836 16785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85712a8afd144249bf425b5e672ba007. Latest consensus state: current_term: 3 leader_uuid: "85712a8afd144249bf425b5e672ba007" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:21.422140 16785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:21.423522 16778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "85712a8afd144249bf425b5e672ba007" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:21.423782 16778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:21.424309 16785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "85712a8afd144249bf425b5e672ba007" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c2e21c0253c4818bc507daf0f09d796" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40395 } } peers { permanent_uuid: "e4144ee7a7954d74a6ecef86288072df" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46699 } attrs { promote: false } } peers { permanent_uuid: "85712a8afd144249bf425b5e672ba007" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 45969 } attrs { promote: false } } }
I20260928 15:03:21.424609 16785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4144ee7a7954d74a6ecef86288072df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:21.440403 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.431792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:34177 (local address 127.0.0.1:55570)
0928 15:03:21.431950 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.431972 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:21.432102 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:21.432647 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:21.432651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:21.432677 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:21.433091 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.433109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.434706 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.434714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.437254 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.437273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.439308 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.439318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.439584 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.439639 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 15:03:21.439755 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:03:21.439942 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 15:03:21.440444 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.431802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.9.12.194:34177)
0928 15:03:21.432246 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.432252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.432318 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:03:21.432404 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.432409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.432506 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.432605 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:21.433281 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.434521 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.437453 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.439078 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.439232 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.439383 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:21.439802 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 15:03:21.440008 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":230,"threads_started":1}
I20260928 15:03:21.441764 16484 tablet_service.cc:1467] Processing CreateTablet for tablet a0150518b4324d27bbc05bb843316e98 (DEFAULT_TABLE table=test-table [id=e60f0e399c3149d09bf0360a7f704cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:21.442238 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0150518b4324d27bbc05bb843316e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:21.447585 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.433447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37065 (local address 127.0.0.1:60840)
0928 15:03:21.441089 (+ 7642us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.441109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:21.441222 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:21.441642 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:21.441650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:21.441685 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:21.441901 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.441914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.443218 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.443222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.444895 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.444908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.446825 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.446831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.447026 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.447058 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:03:21.447140 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:03:21.447271 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7536,"threads_started":1}
I20260928 15:03:21.447764 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.433173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.9.12.193:37065)
0928 15:03:21.436675 (+ 3502us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.436683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.441337 (+ 4654us) server_negotiation.cc:407] Connection header received
0928 15:03:21.441417 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.441422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.441487 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.441610 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:21.442112 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.443075 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.445091 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.446495 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.446625 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.446725 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:21.447332 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 15:03:21.447494 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3351,"thread_start_us":3245,"threads_started":1}
I20260928 15:03:21.449371 16339 tablet_service.cc:1467] Processing CreateTablet for tablet a0150518b4324d27bbc05bb843316e98 (DEFAULT_TABLE table=test-table [id=e60f0e399c3149d09bf0360a7f704cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:21.449508 16796 tablet_bootstrap.cc:492] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Bootstrap starting.
I20260928 15:03:21.449863 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0150518b4324d27bbc05bb843316e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:21.451275 16796 tablet_bootstrap.cc:654] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:21.453138 16796 tablet_bootstrap.cc:492] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: No bootstrap required, opened a new log
I20260928 15:03:21.453322 16796 ts_tablet_manager.cc:1397] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 15:03:21.453972 16796 raft_consensus.cc:348] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.454169 16796 raft_consensus.cc:374] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:21.454282 16796 raft_consensus.cc:729] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b2cbaffa5045f79052d8d1f710a76a, State: Initialized, Role: FOLLOWER
I20260928 15:03:21.454499 16796 consensus_queue.cc:260] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.455266 16796 ts_tablet_manager.cc:1428] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:21.455659 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.433459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.9.12.195:40671)
0928 15:03:21.449074 (+ 15615us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.449080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.449099 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:03:21.449159 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.449163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.449233 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.449331 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:21.450140 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.451111 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.453255 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.454405 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.454487 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.454571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:21.455081 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 15:03:21.455270 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15473,"thread_start_us":15393,"threads_started":1}
I20260928 15:03:21.456782 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.432604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:40671 (local address 127.0.0.1:49156)
0928 15:03:21.433200 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.433215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:21.433315 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:21.449527 (+ 16212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:21.449534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:21.449559 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:21.449941 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.449954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.451279 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.451285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.453088 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.453102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.454705 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.454710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.454872 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.454899 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:03:21.456065 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0928 15:03:21.456292 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":448,"threads_started":1}
I20260928 15:03:21.458806 16549 heartbeater.cc:503] Master 127.9.12.252:45969 was elected leader, sending a full tablet report...
I20260928 15:03:21.458921 16625 tablet_service.cc:1467] Processing CreateTablet for tablet a0150518b4324d27bbc05bb843316e98 (DEFAULT_TABLE table=test-table [id=e60f0e399c3149d09bf0360a7f704cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:21.459443 16625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0150518b4324d27bbc05bb843316e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:21.462941 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.456381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:50665)
0928 15:03:21.462515 (+ 6134us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.462655 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6004,"thread_start_us":5882,"threads_started":1}
I20260928 15:03:21.464457 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.464046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:51289)
0928 15:03:21.464201 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.464314 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:03:21.464932 16799 tablet_bootstrap.cc:492] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Bootstrap starting.
I20260928 15:03:21.466300 16799 tablet_bootstrap.cc:654] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:21.467015 16801 tablet_bootstrap.cc:492] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Bootstrap starting.
I20260928 15:03:21.468415 16799 tablet_bootstrap.cc:492] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: No bootstrap required, opened a new log
I20260928 15:03:21.468576 16799 ts_tablet_manager.cc:1397] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260928 15:03:21.468780 16801 tablet_bootstrap.cc:654] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:21.469405 16799 raft_consensus.cc:348] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.469661 16799 raft_consensus.cc:374] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:21.469756 16799 raft_consensus.cc:729] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a591c02989be4f2c96c8bb60ee820a66, State: Initialized, Role: FOLLOWER
I20260928 15:03:21.470089 16799 consensus_queue.cc:260] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.471006 16801 tablet_bootstrap.cc:492] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: No bootstrap required, opened a new log
I20260928 15:03:21.471186 16801 ts_tablet_manager.cc:1397] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 15:03:21.472200 16801 raft_consensus.cc:348] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.472497 16801 raft_consensus.cc:374] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:21.472800 16799 ts_tablet_manager.cc:1428] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:03:21.474365 16404 heartbeater.cc:503] Master 127.9.12.252:45969 was elected leader, sending a full tablet report...
I20260928 15:03:21.472596 16801 raft_consensus.cc:729] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276bf8b78da443fe9cbbdf6c6c172769, State: Initialized, Role: FOLLOWER
I20260928 15:03:21.475088 16801 consensus_queue.cc:260] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.475924 16801 ts_tablet_manager.cc:1428] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:03:21.476457 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.473003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:41705)
0928 15:03:21.476016 (+ 3013us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.476176 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1870,"thread_start_us":1609,"threads_started":1}
I20260928 15:03:21.478113 16690 heartbeater.cc:503] Master 127.9.12.252:45969 was elected leader, sending a full tablet report...
I20260928 15:03:21.482362 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.477090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:48875)
0928 15:03:21.482135 (+ 5045us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.482199 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4980,"threads_started":1}
I20260928 15:03:21.485412 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.476445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:51317)
0928 15:03:21.485002 (+ 8557us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.485200 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8439,"thread_start_us":8281,"threads_started":1}
I20260928 15:03:21.486229 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.485940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:58939)
0928 15:03:21.486071 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.486133 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 15:03:21.875237 16806 raft_consensus.cc:482] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:21.875461 16806 raft_consensus.cc:504] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.876227 16806 leader_election.cc:288] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177), a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:21.876799 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0150518b4324d27bbc05bb843316e98" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a591c02989be4f2c96c8bb60ee820a66" is_pre_election: true
I20260928 15:03:21.876809 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0150518b4324d27bbc05bb843316e98" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b2cbaffa5045f79052d8d1f710a76a" is_pre_election: true
I20260928 15:03:21.877023 16359 raft_consensus.cc:2454] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 0.
I20260928 15:03:21.877063 16504 raft_consensus.cc:2454] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 0.
I20260928 15:03:21.877525 16580 leader_election.cc:302] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 276bf8b78da443fe9cbbdf6c6c172769, a591c02989be4f2c96c8bb60ee820a66; no voters:
I20260928 15:03:21.877792 16806 raft_consensus.cc:2781] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:21.877888 16806 raft_consensus.cc:482] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:21.877986 16806 raft_consensus.cc:3037] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:21.880232 16806 raft_consensus.cc:504] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.880841 16806 leader_election.cc:288] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 election: Requested vote from peers 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177), a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065)
I20260928 15:03:21.881428 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0150518b4324d27bbc05bb843316e98" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b2cbaffa5045f79052d8d1f710a76a"
I20260928 15:03:21.881439 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0150518b4324d27bbc05bb843316e98" candidate_uuid: "276bf8b78da443fe9cbbdf6c6c172769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a591c02989be4f2c96c8bb60ee820a66"
I20260928 15:03:21.881639 16504 raft_consensus.cc:3037] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:21.881639 16359 raft_consensus.cc:3037] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:21.884548 16359 raft_consensus.cc:2454] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 1.
I20260928 15:03:21.884548 16504 raft_consensus.cc:2454] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276bf8b78da443fe9cbbdf6c6c172769 in term 1.
I20260928 15:03:21.885070 16581 leader_election.cc:302] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 276bf8b78da443fe9cbbdf6c6c172769, 89b2cbaffa5045f79052d8d1f710a76a; no voters:
I20260928 15:03:21.885371 16806 raft_consensus.cc:2781] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:21.885775 16806 raft_consensus.cc:686] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 LEADER]: Becoming Leader. State: Replica: 276bf8b78da443fe9cbbdf6c6c172769, State: Running, Role: LEADER
I20260928 15:03:21.886090 16806 consensus_queue.cc:237] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } }
I20260928 15:03:21.887006 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.886561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:54277)
0928 15:03:21.886714 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.886822 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 15:03:21.888949 16223 catalog_manager.cc:5495] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "276bf8b78da443fe9cbbdf6c6c172769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "276bf8b78da443fe9cbbdf6c6c172769" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 40671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:22.003561 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.998901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.9.12.195:40671)
0928 15:03:21.999529 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.999534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.999550 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:03:21.999612 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.999615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.999681 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.999766 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:22.000107 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.000860 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.002088 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.002479 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.002571 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.002890 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:22.002986 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:22.003051 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:22.003179 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 15:03:22.003240 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":422,"threads_started":1}
I20260928 15:03:22.006699 16504 raft_consensus.cc:1261] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Refusing update from remote peer 276bf8b78da443fe9cbbdf6c6c172769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:22.006729 16359 raft_consensus.cc:1261] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Refusing update from remote peer 276bf8b78da443fe9cbbdf6c6c172769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:22.007370 16806 consensus_queue.cc:1035] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89b2cbaffa5045f79052d8d1f710a76a" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:22.007674 16809 consensus_queue.cc:1035] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a591c02989be4f2c96c8bb60ee820a66" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:22.011224 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.009942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:43269)
0928 15:03:22.010787 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.010928 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":557,"threads_started":1}
I20260928 15:03:22.012485 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.011516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:41577)
0928 15:03:22.012034 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.012182 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":260,"threads_started":1}
I20260928 15:03:22.019178 16818 mvcc.cc:205] Tried to move back new op lower bound from 7334329557013454848 to 7334329556527894528. Current Snapshot: MvccSnapshot[applied={T|T < 7334329557013454848}]
I20260928 15:03:22.022135 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.021697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:34427)
0928 15:03:22.021874 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.021967 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:03:22.023537 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.023120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.195:34629)
0928 15:03:22.023293 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.023380 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:03:22.024082 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.023604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.194:33009)
0928 15:03:22.023846 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.023928 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 15:03:22.026502 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.025946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.9.12.193:58379)
0928 15:03:22.026257 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.026333 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260928 15:03:22.027590 16223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:22.027841 16223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:22.033015 16223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85712a8afd144249bf425b5e672ba007: Sending DeleteTablet for 3 replicas of tablet a0150518b4324d27bbc05bb843316e98
I20260928 15:03:22.033980 16484 tablet_service.cc:1514] Processing DeleteTablet for tablet a0150518b4324d27bbc05bb843316e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:22 UTC) from {username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:55570
I20260928 15:03:22.034292 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet a0150518b4324d27bbc05bb843316e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:22 UTC) from {username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:60840
I20260928 15:03:22.034700 16625 tablet_service.cc:1514] Processing DeleteTablet for tablet a0150518b4324d27bbc05bb843316e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:22 UTC) from {username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:49156
I20260928 15:03:22.034982 16828 tablet_replica.cc:330] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: stopping tablet replica
I20260928 15:03:22.035192 16828 raft_consensus.cc:2229] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:22.035225 16338 tablet_service.cc:1514] Processing DeleteTablet for tablet a0150518b4324d27bbc05bb843316e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:22 UTC) from {username='oryx', principal='oryx/127.9.12.252@KRBTEST.COM'} at 127.0.0.1:60840
I20260928 15:03:22.035511 16828 raft_consensus.cc:2258] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:22.035713 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 16196
I20260928 15:03:22.036422 16828 ts_tablet_manager.cc:1905] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:22.043495 16828 ts_tablet_manager.cc:1918] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:22.043637 16828 log.cc:1196] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/a0150518b4324d27bbc05bb843316e98
I20260928 15:03:22.044143 16828 ts_tablet_manager.cc:1939] T a0150518b4324d27bbc05bb843316e98 P a591c02989be4f2c96c8bb60ee820a66: Deleting consensus metadata
I20260928 15:03:22.046241 16830 tablet_replica.cc:330] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: stopping tablet replica
I20260928 15:03:22.047508 16830 raft_consensus.cc:2229] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:22.048767 16830 raft_consensus.cc:2258] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:22.049448 16827 tablet_replica.cc:330] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: stopping tablet replica
I20260928 15:03:22.049649 16827 raft_consensus.cc:2229] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:22.049917 16827 raft_consensus.cc:2258] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:22.049994 16830 ts_tablet_manager.cc:1905] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:22.050518 16210 catalog_manager.cc:4841] TS a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065): tablet a0150518b4324d27bbc05bb843316e98 (table test-table [id=e60f0e399c3149d09bf0360a7f704cb7]) successfully deleted
I20260928 15:03:22.050825 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.034664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36554)
0928 15:03:22.050460 (+ 15796us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.050575 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15668,"thread_start_us":15544,"threads_started":1}
W20260928 15:03:22.050976 16210 catalog_manager.cc:4808] TS a591c02989be4f2c96c8bb60ee820a66 (127.9.12.193:37065): delete failed for tablet a0150518b4324d27bbc05bb843316e98 because the tablet was not found. No further retry: Not found: Tablet not found: a0150518b4324d27bbc05bb843316e98
I20260928 15:03:22.051205 16827 ts_tablet_manager.cc:1905] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:22.057292 16827 ts_tablet_manager.cc:1918] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:22.057423 16827 log.cc:1196] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/a0150518b4324d27bbc05bb843316e98
I20260928 15:03:22.057847 16827 ts_tablet_manager.cc:1939] T a0150518b4324d27bbc05bb843316e98 P 89b2cbaffa5045f79052d8d1f710a76a: Deleting consensus metadata
I20260928 15:03:22.058468 16830 ts_tablet_manager.cc:1918] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:22.058632 16830 log.cc:1196] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/a0150518b4324d27bbc05bb843316e98
I20260928 15:03:22.058974 16211 catalog_manager.cc:4841] TS 89b2cbaffa5045f79052d8d1f710a76a (127.9.12.194:34177): tablet a0150518b4324d27bbc05bb843316e98 (table test-table [id=e60f0e399c3149d09bf0360a7f704cb7]) successfully deleted
I20260928 15:03:22.059178 16830 ts_tablet_manager.cc:1939] T a0150518b4324d27bbc05bb843316e98 P 276bf8b78da443fe9cbbdf6c6c172769: Deleting consensus metadata
I20260928 15:03:22.060534 16209 catalog_manager.cc:4841] TS 276bf8b78da443fe9cbbdf6c6c172769 (127.9.12.195:40671): tablet a0150518b4324d27bbc05bb843316e98 (table test-table [id=e60f0e399c3149d09bf0360a7f704cb7]) successfully deleted
I20260928 15:03:22.066565 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.066182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36558)
0928 15:03:22.066357 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.066446 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 15:03:22.077021 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.036950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.9.12.193:37065)
0928 15:03:22.044718 (+ 7768us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:22.044727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:22.046781 (+ 2054us) server_negotiation.cc:407] Connection header received
0928 15:03:22.046855 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:22.046862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:22.047011 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:22.047129 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:22.065500 (+ 18371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.066643 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.068521 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.068972 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.069066 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.071652 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:22.071666 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:22.071682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:22.071717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:22.074377 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:22.075025 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:22.075032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:22.075034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:22.075155 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:22.075471 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:22.075474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:22.075475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:22.075859 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:22.076125 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:22.076447 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 15:03:22.076615 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7583,"thread_start_us":7459,"threads_started":1}
I20260928 15:03:22.077702 16399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:22.166208 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 16196
I20260928 15:03:22.183579 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 16416
I20260928 15:03:22.196718 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.184454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.9.12.194:34177)
0928 15:03:22.185033 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:22.185037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:22.185054 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:03:22.185108 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:22.185112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:22.185172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:22.185260 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:22.186615 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.187349 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.188799 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.189235 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.189334 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.191624 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:22.191634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:22.191648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:22.191675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:22.194312 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:22.194923 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:22.194929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:22.194931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:22.195114 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:22.195491 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:22.195494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:22.195496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:22.195819 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:22.196033 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:22.196242 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:03:22.196438 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
I20260928 15:03:22.197317 16544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:22.285496 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 16416
I20260928 15:03:22.302626 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 16562
I20260928 15:03:22.314375 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.303265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.9.12.195:40671)
0928 15:03:22.303465 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:22.303469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:22.303486 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:03:22.303579 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:22.303583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:22.303651 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:22.303726 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:22.304969 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.305698 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.306955 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.307333 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.307425 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.309670 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:22.309684 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:22.309698 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:22.309726 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:22.312191 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:22.312748 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:22.312754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:22.312756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:22.312867 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:22.313152 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:22.313155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:22.313157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:22.313446 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:22.313666 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:22.313864 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 15:03:22.314054 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 15:03:22.314970 16685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:22.432026 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 16562
I20260928 15:03:22.454892 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15894
I20260928 15:03:22.456022 16063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:22.473397 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.472854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36572)
0928 15:03:22.473068 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.473181 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 15:03:22.474377 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.474020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40395 (local address 127.0.0.1:36586)
0928 15:03:22.474160 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.474237 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:03:22.571110 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15894
I20260928 15:03:22.592058 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 15994
I20260928 15:03:22.593130 16254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:22.711238 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 15994
2026-09-28T15:03:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11739 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:22 dist-test-slave-dmhs krb5kdc[16842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:22 dist-test-slave-dmhs krb5kdc[16842](info): set up 2 sockets
Sep 28 15:03:22 dist-test-slave-dmhs krb5kdc[16842](info): commencing operation
krb5kdc: starting...
W20260928 15:03:23.785554 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:03:23 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab.
Entry for principal oryx/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab.
I20260928 15:03:23.993013 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:39749
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab
--principal=oryx/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:39749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:24.306473 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:24.306808 16855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:24.306977 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:24.314014 16855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:24.314143 16855 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:03:24.314245 16855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:24.314348 16855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:24.314429 16855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:24.314518 16855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:24.324173 16855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:39749
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.254.keytab
--principal=oryx/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:39749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:24.325505 16855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:24.326807 16855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:24.338658 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:24.338699 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:24.339200 16855 server_base.cc:1034] running on GCE node
W20260928 15:03:24.339257 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:24.340165 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:24.341430 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:24.342669 16855 hybrid_clock.cc:648] HybridClock initialized: now 1790607804342605 us; error 39 us; skew 500 ppm
I20260928 15:03:24.342995 16855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:24 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607804, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:24.351536 16855 init.cc:377] Logged in from keytab as oryx/127.9.12.254@KRBTEST.COM (short username oryx)
I20260928 15:03:24.353012 16855 webserver.cc:460] Webserver started at http://127.9.12.254:35505/ using document root <none> and password file <none>
I20260928 15:03:24.353662 16855 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:24.353790 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:24.354111 16855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:24.357072 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "a1f8ef6cbbf14cbf84639ad8a6599161"
format_stamp: "Formatted at 2026-09-28 15:03:24 on dist-test-slave-dmhs"
server_key: "d157589abf2132aea0cbcff8fe27dcd1"
server_key_iv: "762b36045db0e96639cd8ca13c7a21a6"
server_key_version: "encryptionkey@0"
I20260928 15:03:24.357867 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "a1f8ef6cbbf14cbf84639ad8a6599161"
format_stamp: "Formatted at 2026-09-28 15:03:24 on dist-test-slave-dmhs"
server_key: "d157589abf2132aea0cbcff8fe27dcd1"
server_key_iv: "762b36045db0e96639cd8ca13c7a21a6"
server_key_version: "encryptionkey@0"
I20260928 15:03:24.362669 16855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:03:24.366705 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:24.368137 16855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:24.368326 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "a1f8ef6cbbf14cbf84639ad8a6599161"
format_stamp: "Formatted at 2026-09-28 15:03:24 on dist-test-slave-dmhs"
server_key: "d157589abf2132aea0cbcff8fe27dcd1"
server_key_iv: "762b36045db0e96639cd8ca13c7a21a6"
server_key_version: "encryptionkey@0"
I20260928 15:03:24.368521 16855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:24.380208 16855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:24.384236 16855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:24.384486 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:24.403636 16855 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:39749
I20260928 15:03:24.403666 16931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:39749 every 8 connection(s)
I20260928 15:03:24.404966 16855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:24.408632 16932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:24.413918 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 16855
I20260928 15:03:24.414122 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:24.414770 9267 external_mini_cluster.cc:1428] Setting key fb7d72b0950b18848ae1e5d2d40df6fb
I20260928 15:03:24.418771 16932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Bootstrap starting.
I20260928 15:03:24.422322 16932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:24.423333 16932 log.cc:824] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:24.426229 16932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: No bootstrap required, opened a new log
Sep 28 15:03:24 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:24.434567 16932 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39749 } }
I20260928 15:03:24.434928 16932 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:24.435063 16932 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f8ef6cbbf14cbf84639ad8a6599161, State: Initialized, Role: FOLLOWER
I20260928 15:03:24.435775 16932 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39749 } }
I20260928 15:03:24.436079 16932 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:24.436214 16932 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:24.436476 16932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:24.437361 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.416750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.9.12.254:39749)
0928 15:03:24.417606 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:24.417617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:24.417656 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:03:24.419416 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:24.419438 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:24.419818 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:24.420134 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:24.421897 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.423177 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.424744 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.425383 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.425508 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.430060 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:24.430067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:24.430080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:24.430111 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:24.433099 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:24.433774 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:24.433780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:24.433786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:24.433944 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:24.434333 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:24.434337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:24.434339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:24.434796 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:24.435107 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:24.435548 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 15:03:24.435997 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":380,"threads_started":1}
I20260928 15:03:24.438987 16932 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39749 } }
I20260928 15:03:24.439443 16932 leader_election.cc:302] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f8ef6cbbf14cbf84639ad8a6599161; no voters:
I20260928 15:03:24.440042 16932 leader_election.cc:288] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:24.440176 16937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:24.440899 16937 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [term 1 LEADER]: Becoming Leader. State: Replica: a1f8ef6cbbf14cbf84639ad8a6599161, State: Running, Role: LEADER
I20260928 15:03:24.441411 16937 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39749 } }
I20260928 15:03:24.442036 16932 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:24.443912 16938 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39749 } } }
I20260928 15:03:24.444357 16938 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:24.444015 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f8ef6cbbf14cbf84639ad8a6599161. Latest consensus state: current_term: 1 leader_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f8ef6cbbf14cbf84639ad8a6599161" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39749 } } }
I20260928 15:03:24.444653 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:24.445312 16945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:24.461476 16945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:24.469892 16945 catalog_manager.cc:1347] Generated new cluster ID: ee84f359f5634d3bb87849f32eede4f5
I20260928 15:03:24.470019 16945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:24.485639 16945 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:24.487128 16945 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:24.505264 16945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Generated new TSK 0
I20260928 15:03:24.506206 16945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab.
Entry for principal oryx/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab.
I20260928 15:03:24.565589 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:39749
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:24.882380 16958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:24.882675 16958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:24.882827 16958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:24.889786 16958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:24.889935 16958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:24.890089 16958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:03:24.890228 16958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:24.899490 16958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:39749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:24.900707 16958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:24.901958 16958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:24.913964 16963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:24.914031 16964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:24.914006 16966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:24.914728 16958 server_base.cc:1034] running on GCE node
I20260928 15:03:24.915223 16958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:24.916056 16958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:24.917256 16958 hybrid_clock.cc:648] HybridClock initialized: now 1790607804917168 us; error 61 us; skew 500 ppm
I20260928 15:03:24.917480 16958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:24 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607804, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:24.920481 16958 init.cc:377] Logged in from keytab as oryx/127.9.12.193@KRBTEST.COM (short username oryx)
I20260928 15:03:24.921765 16958 webserver.cc:460] Webserver started at http://127.9.12.193:42309/ using document root <none> and password file <none>
I20260928 15:03:24.922430 16958 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:24.922556 16958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:24.922870 16958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:24.925710 16958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "9fe519763ff741a7b3b60e50233f5892"
format_stamp: "Formatted at 2026-09-28 15:03:24 on dist-test-slave-dmhs"
server_key: "3c0713dbe5e9a2f119d6a8d967212c13"
server_key_iv: "62927c20a67398b10950d21a17fd9641"
server_key_version: "encryptionkey@0"
I20260928 15:03:24.926501 16958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "9fe519763ff741a7b3b60e50233f5892"
format_stamp: "Formatted at 2026-09-28 15:03:24 on dist-test-slave-dmhs"
server_key: "3c0713dbe5e9a2f119d6a8d967212c13"
server_key_iv: "62927c20a67398b10950d21a17fd9641"
server_key_version: "encryptionkey@0"
I20260928 15:03:24.931354 16958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 15:03:24.934679 16973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:24.935878 16958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:24.936048 16958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "9fe519763ff741a7b3b60e50233f5892"
format_stamp: "Formatted at 2026-09-28 15:03:24 on dist-test-slave-dmhs"
server_key: "3c0713dbe5e9a2f119d6a8d967212c13"
server_key_iv: "62927c20a67398b10950d21a17fd9641"
server_key_version: "encryptionkey@0"
I20260928 15:03:24.936340 16958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:24.950100 16958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:24.953910 16958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:24.954131 16958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:24.956949 16958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:24.957072 16958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:24.957232 16958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:24.957302 16958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:03:24 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607804, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:24.980556 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.957377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:34407 (local address 127.9.12.254:39749)
0928 15:03:24.959739 (+ 2362us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:24.959746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:24.959774 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:03:24.961468 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:24.961474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:24.961583 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:24.961729 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:24.964003 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.964992 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.967458 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.967860 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.968013 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.972771 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:24.972780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:24.972788 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:24.972838 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:24.976774 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:24.977430 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:24.977438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:24.977444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:24.977604 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:24.978182 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:24.978188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:24.978193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:24.978607 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:24.978755 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:24.979962 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0928 15:03:24.980168 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2229,"thread_start_us":2112,"threads_started":1}
I20260928 15:03:24.981693 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.958008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.9.12.193:34407)
0928 15:03:24.959056 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:24.959129 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:24.961200 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:24.962047 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:24.962058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:24.962848 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:24.963765 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.963792 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.965127 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.965134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.967319 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.967331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.968094 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.968103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.968452 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.969278 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:24.969312 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:24.972552 (+ 3240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:24.976936 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:24.976942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:24.976952 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:24.977321 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:24.977761 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:24.977768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:24.977773 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:24.978059 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:24.978743 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:24.978750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:24.979260 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 15:03:24.979588 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 15:03:24.980159 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":377,"threads_started":1}
I20260928 15:03:25.006491 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.990462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:52957 (local address 127.9.12.254:39749)
0928 15:03:24.990661 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:24.990669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:24.991637 (+ 968us) server_negotiation.cc:407] Connection header received
0928 15:03:24.991751 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:24.991759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:24.991851 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:24.991966 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:24.993048 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.994252 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.995529 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.996325 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.996498 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.999299 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:24.999308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:24.999315 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:24.999360 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:25.002889 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.004038 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.004045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.004050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.004207 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.004698 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.004705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.004710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.005118 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:25.005334 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:25.006065 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 15:03:25.006247 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:03:25.007175 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.990469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.9.12.193:52957)
0928 15:03:24.991519 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:24.991538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:24.991654 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:24.992005 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:24.992009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:24.992617 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:24.992901 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.992909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.994413 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.994422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.995391 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.995402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.996566 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.996574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.996839 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.997575 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:24.997613 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:24.999106 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:25.003075 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.003085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.003092 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.003897 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.004366 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.004374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.004380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.004559 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.005296 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:25.005304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:25.005655 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 15:03:25.005922 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 15:03:25.006383 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":772,"threads_started":1}
I20260928 15:03:25.011106 16958 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:38255
I20260928 15:03:25.011137 17093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:38255 every 8 connection(s)
I20260928 15:03:25.012349 16958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:25.014261 17094 heartbeater.cc:347] Connected to a master server at 127.9.12.254:39749
I20260928 15:03:25.014438 17094 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:25.014988 17094 heartbeater.cc:511] Master 127.9.12.254:39749 requested a full tablet report, sending...
I20260928 15:03:25.016851 16887 ts_manager.cc:194] Registered new tserver with Master: 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255)
I20260928 15:03:25.018131 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 16958
I20260928 15:03:25.018316 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:25.018728 16887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:34407
I20260928 15:03:25.019146 9267 external_mini_cluster.cc:1428] Setting key 162d39f1cfc388db33fc82f34d0b0639
WARNING: no policy specified for oryx/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab.
Entry for principal oryx/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab.
I20260928 15:03:25.063134 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:39749
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:25.371881 17098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:25.372128 17098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:25.372334 17098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:25.379182 17098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:25.379272 17098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:25.379419 17098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:03:25.379546 17098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:25.389509 17098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:39749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:25.390729 17098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:25.391996 17098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:25.404649 17104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:25.404681 17103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:25.405335 17106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:25.405539 17098 server_base.cc:1034] running on GCE node
I20260928 15:03:25.406173 17098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:25.407095 17098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:25.408366 17098 hybrid_clock.cc:648] HybridClock initialized: now 1790607805408286 us; error 48 us; skew 500 ppm
I20260928 15:03:25.408651 17098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:25 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607805, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:25.411540 17098 init.cc:377] Logged in from keytab as oryx/127.9.12.194@KRBTEST.COM (short username oryx)
I20260928 15:03:25.413012 17098 webserver.cc:460] Webserver started at http://127.9.12.194:38605/ using document root <none> and password file <none>
I20260928 15:03:25.413616 17098 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:25.413697 17098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:25.414033 17098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:25.416857 17098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "d42fd6f340854d89b0d6d82683302b00"
format_stamp: "Formatted at 2026-09-28 15:03:25 on dist-test-slave-dmhs"
server_key: "47649bbb0f43b9fbcbbf52534cbe3759"
server_key_iv: "7ea8c3814b35a3dc3c77d9e1edd984b1"
server_key_version: "encryptionkey@0"
I20260928 15:03:25.417611 17098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "d42fd6f340854d89b0d6d82683302b00"
format_stamp: "Formatted at 2026-09-28 15:03:25 on dist-test-slave-dmhs"
server_key: "47649bbb0f43b9fbcbbf52534cbe3759"
server_key_iv: "7ea8c3814b35a3dc3c77d9e1edd984b1"
server_key_version: "encryptionkey@0"
I20260928 15:03:25.422349 17098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:03:25.425513 17113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:25.426651 17098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:25.426798 17098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "d42fd6f340854d89b0d6d82683302b00"
format_stamp: "Formatted at 2026-09-28 15:03:25 on dist-test-slave-dmhs"
server_key: "47649bbb0f43b9fbcbbf52534cbe3759"
server_key_iv: "7ea8c3814b35a3dc3c77d9e1edd984b1"
server_key_version: "encryptionkey@0"
I20260928 15:03:25.427052 17098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:25.446507 17098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:25.450630 17098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:25.450892 17098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:25.453372 17098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:25.453450 17098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:25.453632 17098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:25.453691 17098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:03:25 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607805, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:25.477061 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.454004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:56107 (local address 127.9.12.254:39749)
0928 15:03:25.454212 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.454217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.455622 (+ 1405us) server_negotiation.cc:407] Connection header received
0928 15:03:25.457971 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.457979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.458095 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.458182 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.460402 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.461678 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.464064 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.464676 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.464806 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.469673 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:25.469678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:25.469681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:25.469710 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:25.473052 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.473934 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.473943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.473947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.474088 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.474732 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.474740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.474746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.475167 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:25.475384 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:25.476236 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 15:03:25.476585 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:03:25.478439 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.454525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.9.12.194:56107)
0928 15:03:25.455445 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.455500 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.457630 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.458664 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.458674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.459377 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.460196 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.460228 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.461860 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.461872 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.463877 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.463895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.464924 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.464930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.465220 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.466077 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.466121 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.469408 (+ 3287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:25.473176 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.473187 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.473202 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.473811 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.474204 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.474209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.474213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.474519 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.475345 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:25.475358 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:25.475799 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 15:03:25.476097 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 15:03:25.476789 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":380,"threads_started":1}
I20260928 15:03:25.498718 17098 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:35723
I20260928 15:03:25.498835 17232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:35723 every 8 connection(s)
I20260928 15:03:25.500329 17098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:25.502434 17233 heartbeater.cc:347] Connected to a master server at 127.9.12.254:39749
I20260928 15:03:25.502835 17233 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:25.503536 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 17098
I20260928 15:03:25.503602 17233 heartbeater.cc:511] Master 127.9.12.254:39749 requested a full tablet report, sending...
I20260928 15:03:25.503708 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:25.504303 9267 external_mini_cluster.cc:1428] Setting key 6d4eb191256993d1e195787966941d73
I20260928 15:03:25.504944 16887 ts_manager.cc:194] Registered new tserver with Master: d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723)
I20260928 15:03:25.506171 16887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:56107
I20260928 15:03:25.508019 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.491879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:40411 (local address 127.9.12.254:39749)
0928 15:03:25.492059 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.492063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.493002 (+ 939us) server_negotiation.cc:407] Connection header received
0928 15:03:25.493210 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.493217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.493316 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.493442 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.495190 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.496273 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.498533 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.499150 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.499243 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.501573 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:25.501579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:25.501584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:25.501626 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:25.504984 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.505784 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.505791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.505796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.505970 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.506413 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.506417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.506420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.506827 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:25.507002 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:25.507494 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 15:03:25.507711 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:03:25.508652 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.492018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.9.12.194:40411)
0928 15:03:25.492864 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.492890 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.493065 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.493687 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.493693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.494441 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.494996 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.495017 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.496455 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.496464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.498359 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.498374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.499323 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.499330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.499513 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.500102 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.500123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.501343 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:25.505135 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.505140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.505143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.505648 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.506100 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.506106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.506111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.506276 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.507008 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:25.507014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:25.507204 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 15:03:25.507375 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 15:03:25.508040 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":47,"thread_start_us":463,"threads_started":1}
WARNING: no policy specified for oryx/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab.
Entry for principal oryx/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab.
I20260928 15:03:25.551199 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:39749
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:25.866950 17237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:25.867226 17237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:25.867326 17237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:25.874320 17237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:25.874413 17237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:25.874495 17237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:03:25.874572 17237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:25.883641 17237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.12.254:39749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:25.884866 17237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:25.886085 17237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:25.898693 17242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:25.898797 17243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:25.898859 17245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:25.899791 17237 server_base.cc:1034] running on GCE node
I20260928 15:03:25.900462 17237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:25.901347 17237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:25.902554 17237 hybrid_clock.cc:648] HybridClock initialized: now 1790607805902475 us; error 50 us; skew 500 ppm
I20260928 15:03:25.902838 17237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:25 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607805, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:25.906020 17237 init.cc:377] Logged in from keytab as oryx/127.9.12.195@KRBTEST.COM (short username oryx)
I20260928 15:03:25.907457 17237 webserver.cc:460] Webserver started at http://127.9.12.195:37311/ using document root <none> and password file <none>
I20260928 15:03:25.908141 17237 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:25.908250 17237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:25.908604 17237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:25.911422 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
format_stamp: "Formatted at 2026-09-28 15:03:25 on dist-test-slave-dmhs"
server_key: "b7d56f7036293d68325d90aeb6595856"
server_key_iv: "3378df4b019c221a34cbe7d6773bc3f3"
server_key_version: "encryptionkey@0"
I20260928 15:03:25.912082 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
format_stamp: "Formatted at 2026-09-28 15:03:25 on dist-test-slave-dmhs"
server_key: "b7d56f7036293d68325d90aeb6595856"
server_key_iv: "3378df4b019c221a34cbe7d6773bc3f3"
server_key_version: "encryptionkey@0"
I20260928 15:03:25.916888 17237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:03:25.920115 17252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:25.921375 17237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 15:03:25.921541 17237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
format_stamp: "Formatted at 2026-09-28 15:03:25 on dist-test-slave-dmhs"
server_key: "b7d56f7036293d68325d90aeb6595856"
server_key_iv: "3378df4b019c221a34cbe7d6773bc3f3"
server_key_version: "encryptionkey@0"
I20260928 15:03:25.921761 17237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:25.942260 17237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:25.946385 17237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:25.946635 17237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:25.949218 17237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:25.949299 17237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:25.949488 17237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:25.949570 17237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:03:25 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607805, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:25.975063 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.949766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:49969 (local address 127.9.12.254:39749)
0928 15:03:25.949932 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.949939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.951582 (+ 1643us) server_negotiation.cc:407] Connection header received
0928 15:03:25.954151 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.954161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.954285 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.954410 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.956869 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.958027 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.960702 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.961315 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.961470 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.966802 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:25.966807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:25.966810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:25.966845 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:25.970622 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.971651 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.971656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.971659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.971862 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.972607 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.972613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.972618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.973016 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:25.973176 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:25.974195 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0928 15:03:25.974513 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:03:25.975823 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.950626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.9.12.195:49969)
0928 15:03:25.951415 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.951459 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.953728 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.954742 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.954753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.955637 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.956627 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.956654 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.958221 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.958225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.960496 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.960515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.961587 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.961595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.961912 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.962802 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.962843 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.966473 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:25.970762 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.970771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.970791 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.971518 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.972025 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.972032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.972037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.972403 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.973152 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:25.973163 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:25.973702 (+ 539us) client_negotiation.cc:769] Sending connection context
0928 15:03:25.974047 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 15:03:25.974508 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":351,"threads_started":1}
I20260928 15:03:25.995582 17237 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:45265
I20260928 15:03:25.995608 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:45265 every 8 connection(s)
I20260928 15:03:25.996866 17237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:25.997697 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.982270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:48331 (local address 127.9.12.254:39749)
0928 15:03:25.982485 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.982492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.983168 (+ 676us) server_negotiation.cc:407] Connection header received
0928 15:03:25.983358 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.983361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.983456 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.983548 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.985195 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.986227 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.988497 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.988958 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.989091 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.991757 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:25.991765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:25.991770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:25.991814 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:25.994887 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.995615 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.995622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.995625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.995803 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:25.996180 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:25.996184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:25.996188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:25.996529 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:25.996697 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:25.997091 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 15:03:25.997318 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:03:25.997809 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.982274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.9.12.195:48331)
0928 15:03:25.983041 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.983071 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.983239 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.983607 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.983615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.984383 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.984954 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.984972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.986333 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.986338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.988343 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.988357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.989401 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.989413 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.989659 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.990369 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.990396 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.991527 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:25.995008 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.995014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.995018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.995476 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.995930 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:25.995935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:25.995939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:25.996069 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:25.996657 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:25.996662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:25.996841 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 15:03:25.996996 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 15:03:25.997340 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":385,"threads_started":1}
I20260928 15:03:25.999208 17372 heartbeater.cc:347] Connected to a master server at 127.9.12.254:39749
I20260928 15:03:25.999745 17372 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:26.000146 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 17237
I20260928 15:03:26.000341 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:26.000547 17372 heartbeater.cc:511] Master 127.9.12.254:39749 requested a full tablet report, sending...
I20260928 15:03:26.000919 9267 external_mini_cluster.cc:1428] Setting key 9dff455a1c0317421877ba849c73727c
I20260928 15:03:26.001932 16887 ts_manager.cc:194] Registered new tserver with Master: 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:26.002851 16887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:49969
I20260928 15:03:26.004032 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:26.021543 17094 heartbeater.cc:503] Master 127.9.12.254:39749 was elected leader, sending a full tablet report...
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:26.052521 16982 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:26.042545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.9.12.254:39749)
0928 15:03:26.042753 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.042760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.042879 (+ 119us) server_negotiation.cc:407] Connection header received
0928 15:03:26.043138 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.043143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.043222 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.043331 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.044628 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.045283 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.046451 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.046898 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.047016 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.052229 (+ 5213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38270: BlockingRecv error: recv got EOF from 127.0.0.1:38270 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:26.068220 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.055298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.9.12.254:39749)
0928 15:03:26.055520 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.055528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.055855 (+ 327us) server_negotiation.cc:407] Connection header received
0928 15:03:26.056055 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.056059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.056118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.056220 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.057346 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.058026 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.059218 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.059662 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.059817 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.063367 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:26.063372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:26.063376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:26.063406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:26.066002 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:26.066580 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:26.066584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:26.066586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:26.066708 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:26.067050 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:26.067053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:26.067055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:26.067294 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:26.067506 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.067711 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.067876 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:03:26.072024 16886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:26.078833 16886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:26.105417 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.092581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.9.12.194:35723)
0928 15:03:26.097991 (+ 5410us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.098004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.098027 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:03:26.098099 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.098104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.098283 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.098472 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.099107 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.100604 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.103090 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.104338 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.104479 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.104693 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.104958 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.105116 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5266,"thread_start_us":5157,"threads_started":1}
I20260928 15:03:26.105485 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.092243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:35723 (local address 127.0.0.1:48722)
0928 15:03:26.097104 (+ 4861us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.097127 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.097259 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.098474 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.098481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.098508 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.098871 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.098885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.100765 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.100770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.102937 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.102960 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.104544 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.104550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.104734 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.104784 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.104916 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.105102 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4712,"mutex_wait_us":1,"thread_start_us":4583,"threads_started":1}
I20260928 15:03:26.106532 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.090816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:45265 (local address 127.0.0.1:33844)
0928 15:03:26.091631 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.091673 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.091848 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.099941 (+ 8093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.099946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.099970 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.100648 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.100663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.102157 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.102163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.103894 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.103907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.105738 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.105744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.105908 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.105944 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.106018 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.106142 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":23,"spinlock_wait_cycles":11520,"thread_start_us":774,"threads_started":1}
I20260928 15:03:26.107892 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.091383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.9.12.195:45265)
0928 15:03:26.099179 (+ 7796us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.099187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.099217 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:03:26.099321 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.099328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.099573 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.100369 (+ 796us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.100844 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.102039 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.104074 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.105514 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.107099 (+ 1585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.107344 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.107411 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.107577 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7627,"thread_start_us":7452,"threads_started":1}
I20260928 15:03:26.108129 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.091352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.9.12.193:38255)
0928 15:03:26.099505 (+ 8153us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.099511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.099544 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:03:26.099611 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.099641 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.099840 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.100036 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.101407 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.102809 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.105184 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.106685 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.106809 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.107055 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.107702 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.107851 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8010,"thread_start_us":7886,"threads_started":1}
I20260928 15:03:26.108403 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.090867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38255 (local address 127.0.0.1:40780)
0928 15:03:26.091631 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.091670 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.091847 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.100171 (+ 8324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.100175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.100199 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.101200 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.101213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.102946 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.102950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.104995 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.105010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.106896 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.106902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.107108 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.107165 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.107259 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.107375 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":460,"spinlock_wait_cycles":1792,"thread_start_us":624,"threads_started":1}
I20260928 15:03:26.109450 17163 tablet_service.cc:1467] Processing CreateTablet for tablet f500bcbe767e4fd98d23563049aa6d01 (DEFAULT_TABLE table=test-table [id=9fceb8f6b32d431ebccc327bbf8cc909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:26.110873 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f500bcbe767e4fd98d23563049aa6d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.111395 17302 tablet_service.cc:1467] Processing CreateTablet for tablet f500bcbe767e4fd98d23563049aa6d01 (DEFAULT_TABLE table=test-table [id=9fceb8f6b32d431ebccc327bbf8cc909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:26.112730 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f500bcbe767e4fd98d23563049aa6d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.112972 17024 tablet_service.cc:1467] Processing CreateTablet for tablet f500bcbe767e4fd98d23563049aa6d01 (DEFAULT_TABLE table=test-table [id=9fceb8f6b32d431ebccc327bbf8cc909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:26.114005 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f500bcbe767e4fd98d23563049aa6d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.123879 17396 tablet_bootstrap.cc:492] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Bootstrap starting.
I20260928 15:03:26.125569 17397 tablet_bootstrap.cc:492] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Bootstrap starting.
I20260928 15:03:26.128932 17398 tablet_bootstrap.cc:492] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Bootstrap starting.
I20260928 15:03:26.128991 17396 tablet_bootstrap.cc:654] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.128988 17397 tablet_bootstrap.cc:654] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.130223 17396 log.cc:824] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:26.130223 17397 log.cc:824] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:26.132503 17398 tablet_bootstrap.cc:654] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.133044 17397 tablet_bootstrap.cc:492] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: No bootstrap required, opened a new log
I20260928 15:03:26.133282 17397 ts_tablet_manager.cc:1397] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:03:26.133503 17396 tablet_bootstrap.cc:492] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: No bootstrap required, opened a new log
I20260928 15:03:26.133658 17398 log.cc:824] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:26.133831 17396 ts_tablet_manager.cc:1397] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 15:03:26.141580 17397 raft_consensus.cc:348] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.141945 17397 raft_consensus.cc:374] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.142067 17397 raft_consensus.cc:729] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.142692 17397 consensus_queue.cc:260] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.142910 17398 tablet_bootstrap.cc:492] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: No bootstrap required, opened a new log
I20260928 15:03:26.143149 17398 ts_tablet_manager.cc:1397] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260928 15:03:26.143884 17233 heartbeater.cc:503] Master 127.9.12.254:39749 was elected leader, sending a full tablet report...
I20260928 15:03:26.144560 17397 ts_tablet_manager.cc:1428] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 15:03:26.147097 17396 raft_consensus.cc:348] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.147449 17396 raft_consensus.cc:374] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.147567 17396 raft_consensus.cc:729] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.148176 17396 consensus_queue.cc:260] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.149642 17396 ts_tablet_manager.cc:1428] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260928 15:03:26.153314 17398 raft_consensus.cc:348] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.153693 17398 raft_consensus.cc:374] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.153789 17398 raft_consensus.cc:729] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca6626cc1f54bb78745432a42d2fbdb, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.154299 17398 consensus_queue.cc:260] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.155224 17372 heartbeater.cc:503] Master 127.9.12.254:39749 was elected leader, sending a full tablet report...
I20260928 15:03:26.155726 17398 ts_tablet_manager.cc:1428] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20260928 15:03:26.249209 17373 tablet.cc:2369] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:26.252449 17234 tablet.cc:2369] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:26.264715 17095 tablet.cc:2369] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:26.303308 17402 raft_consensus.cc:482] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:26.303567 17402 raft_consensus.cc:504] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.305104 17402 leader_election.cc:288] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265), 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255)
I20260928 15:03:26.313695 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.305678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:45463 (local address 127.9.12.193:38255)
0928 15:03:26.305865 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.305873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.306470 (+ 597us) server_negotiation.cc:407] Connection header received
0928 15:03:26.306684 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.306689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.306772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.306856 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.307751 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.308884 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.311552 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.312691 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.312833 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.312897 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.313364 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.313475 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:03:26.313695 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.305581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:35461 (local address 127.9.12.195:45265)
0928 15:03:26.305784 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.305790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.306493 (+ 703us) server_negotiation.cc:407] Connection header received
0928 15:03:26.306684 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.306689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.306763 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.306856 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.307820 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.308894 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.311553 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.312695 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.312837 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.312897 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.313364 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.313475 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:03:26.313897 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.305414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38255 (local address 127.9.12.194:45463)
0928 15:03:26.306318 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.306339 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.306511 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.306914 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.306918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.306989 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.307554 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.307583 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.309018 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.309023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.311401 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.311414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.312933 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.312937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.313166 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.313244 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.313328 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.313481 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":277,"spinlock_wait_cycles":13824,"thread_start_us":650,"threads_started":1}
I20260928 15:03:26.313920 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.305518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:45265 (local address 127.9.12.194:35461)
0928 15:03:26.306380 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.306396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.306518 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.306914 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.306918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.306990 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.307638 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.307650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.309019 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.309023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.311401 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.311414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.312930 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.312935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.313165 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.313241 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.313328 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.313481 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":248,"spinlock_wait_cycles":68352,"thread_start_us":433,"threads_started":1}
I20260928 15:03:26.314924 17322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f500bcbe767e4fd98d23563049aa6d01" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" is_pre_election: true
I20260928 15:03:26.315371 17322 raft_consensus.cc:2454] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 0.
I20260928 15:03:26.315999 17115 leader_election.cc:302] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:26.316169 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f500bcbe767e4fd98d23563049aa6d01" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892" is_pre_election: true
I20260928 15:03:26.316428 17402 raft_consensus.cc:2781] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:26.316632 17402 raft_consensus.cc:482] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:26.316736 17402 raft_consensus.cc:3037] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.316740 17049 raft_consensus.cc:2454] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 0.
I20260928 15:03:26.318934 17402 raft_consensus.cc:504] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.319522 17402 leader_election.cc:288] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 election: Requested vote from peers 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265), 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255)
I20260928 15:03:26.319957 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f500bcbe767e4fd98d23563049aa6d01" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892"
I20260928 15:03:26.319963 17322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f500bcbe767e4fd98d23563049aa6d01" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
I20260928 15:03:26.320155 17049 raft_consensus.cc:3037] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.320163 17322 raft_consensus.cc:3037] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.322197 17049 raft_consensus.cc:2454] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 1.
I20260928 15:03:26.322207 17322 raft_consensus.cc:2454] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 1.
I20260928 15:03:26.322614 17114 leader_election.cc:302] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe519763ff741a7b3b60e50233f5892, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:26.322881 17402 raft_consensus.cc:2781] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:26.323289 17402 raft_consensus.cc:686] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 1 LEADER]: Becoming Leader. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Running, Role: LEADER
I20260928 15:03:26.323797 17402 consensus_queue.cc:237] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:26.328336 16886 catalog_manager.cc:5495] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to d42fd6f340854d89b0d6d82683302b00 (127.9.12.194). New cstate: current_term: 1 leader_uuid: "d42fd6f340854d89b0d6d82683302b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } health_report { overall_health: HEALTHY } } }
I20260928 15:03:26.398947 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.394091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.9.12.194:35723)
0928 15:03:26.394274 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.394279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.394437 (+ 158us) server_negotiation.cc:407] Connection header received
0928 15:03:26.394624 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.394628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.394714 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.394834 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:26.395132 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.395844 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.397180 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.397570 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.397660 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.397995 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.398499 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.398613 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.398688 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.398752 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:03:26.406626 17049 raft_consensus.cc:1261] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Refusing update from remote peer d42fd6f340854d89b0d6d82683302b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:26.406639 17322 raft_consensus.cc:1261] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Refusing update from remote peer d42fd6f340854d89b0d6d82683302b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:26.407685 17407 consensus_queue.cc:1035] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:26.408067 17402 consensus_queue.cc:1035] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:26.423513 17412 mvcc.cc:205] Tried to move back new op lower bound from 7334329575031615488 to 7334329574706393088. Current Snapshot: MvccSnapshot[applied={T|T < 7334329575031615488}]
I20260928 15:03:26.440656 16887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:26.440932 16887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:26.444626 16887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Sending DeleteTablet for 3 replicas of tablet f500bcbe767e4fd98d23563049aa6d01
I20260928 15:03:26.445885 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet f500bcbe767e4fd98d23563049aa6d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:26 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:33844
I20260928 15:03:26.446800 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet f500bcbe767e4fd98d23563049aa6d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:26 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48722
I20260928 15:03:26.447033 17419 tablet_replica.cc:330] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: stopping tablet replica
I20260928 15:03:26.447501 17419 raft_consensus.cc:2229] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:26.448024 17419 raft_consensus.cc:2258] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:26.448038 17420 tablet_replica.cc:330] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: stopping tablet replica
I20260928 15:03:26.448693 17420 raft_consensus.cc:2229] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:26.449409 17420 raft_consensus.cc:2258] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:26.450467 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet f500bcbe767e4fd98d23563049aa6d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:26 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:40780
I20260928 15:03:26.451440 17420 ts_tablet_manager.cc:1905] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:26.451557 17421 tablet_replica.cc:330] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: stopping tablet replica
I20260928 15:03:26.452049 17421 raft_consensus.cc:2229] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:26.452868 17421 raft_consensus.cc:2258] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:26.453454 16887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:26.454190 16887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:26.457182 17419 ts_tablet_manager.cc:1905] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:26.458877 17420 ts_tablet_manager.cc:1918] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:26.459066 17420 log.cc:1196] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/f500bcbe767e4fd98d23563049aa6d01
I20260928 15:03:26.459630 17420 ts_tablet_manager.cc:1939] T f500bcbe767e4fd98d23563049aa6d01 P d42fd6f340854d89b0d6d82683302b00: Deleting consensus metadata
I20260928 15:03:26.461565 16872 catalog_manager.cc:4841] TS d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723): tablet f500bcbe767e4fd98d23563049aa6d01 (table test-table [id=9fceb8f6b32d431ebccc327bbf8cc909]) successfully deleted
I20260928 15:03:26.462234 17024 tablet_service.cc:1467] Processing CreateTablet for tablet ee938f803846463ab68e78530384f4b8 (DEFAULT_TABLE table=test-table [id=3bd952f0e92f446fa6b105d5a2ebe01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:26.462479 17302 tablet_service.cc:1467] Processing CreateTablet for tablet ee938f803846463ab68e78530384f4b8 (DEFAULT_TABLE table=test-table [id=3bd952f0e92f446fa6b105d5a2ebe01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:26.462673 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee938f803846463ab68e78530384f4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.462899 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee938f803846463ab68e78530384f4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.463865 17419 ts_tablet_manager.cc:1918] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:26.464012 17419 log.cc:1196] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/f500bcbe767e4fd98d23563049aa6d01
I20260928 15:03:26.464536 17419 ts_tablet_manager.cc:1939] T f500bcbe767e4fd98d23563049aa6d01 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting consensus metadata
I20260928 15:03:26.466598 17163 tablet_service.cc:1467] Processing CreateTablet for tablet ee938f803846463ab68e78530384f4b8 (DEFAULT_TABLE table=test-table [id=3bd952f0e92f446fa6b105d5a2ebe01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:26.467043 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee938f803846463ab68e78530384f4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.467546 17421 ts_tablet_manager.cc:1905] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:26.473527 16872 catalog_manager.cc:4841] TS 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265): tablet f500bcbe767e4fd98d23563049aa6d01 (table test-table [id=9fceb8f6b32d431ebccc327bbf8cc909]) successfully deleted
I20260928 15:03:26.474378 17396 tablet_bootstrap.cc:492] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Bootstrap starting.
I20260928 15:03:26.475724 17396 tablet_bootstrap.cc:654] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.477113 17398 tablet_bootstrap.cc:492] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Bootstrap starting.
I20260928 15:03:26.474354 17421 ts_tablet_manager.cc:1918] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:26.477456 17421 log.cc:1196] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/f500bcbe767e4fd98d23563049aa6d01
I20260928 15:03:26.478060 17421 ts_tablet_manager.cc:1939] T f500bcbe767e4fd98d23563049aa6d01 P 9fe519763ff741a7b3b60e50233f5892: Deleting consensus metadata
I20260928 15:03:26.478653 17396 tablet_bootstrap.cc:492] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: No bootstrap required, opened a new log
I20260928 15:03:26.478662 17398 tablet_bootstrap.cc:654] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.478835 17396 ts_tablet_manager.cc:1397] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 15:03:26.479892 16871 catalog_manager.cc:4841] TS 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255): tablet f500bcbe767e4fd98d23563049aa6d01 (table test-table [id=9fceb8f6b32d431ebccc327bbf8cc909]) successfully deleted
I20260928 15:03:26.480669 17396 raft_consensus.cc:348] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.480916 17396 raft_consensus.cc:374] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.481041 17396 raft_consensus.cc:729] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.481349 17396 consensus_queue.cc:260] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.481798 17397 tablet_bootstrap.cc:492] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Bootstrap starting.
I20260928 15:03:26.481966 17396 ts_tablet_manager.cc:1428] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:26.483516 17397 tablet_bootstrap.cc:654] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.489909 17398 tablet_bootstrap.cc:492] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: No bootstrap required, opened a new log
I20260928 15:03:26.490065 17398 ts_tablet_manager.cc:1397] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20260928 15:03:26.490675 17397 tablet_bootstrap.cc:492] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: No bootstrap required, opened a new log
I20260928 15:03:26.490846 17397 ts_tablet_manager.cc:1397] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260928 15:03:26.490959 17398 raft_consensus.cc:348] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.491200 17398 raft_consensus.cc:374] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.491297 17398 raft_consensus.cc:729] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca6626cc1f54bb78745432a42d2fbdb, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.491570 17398 consensus_queue.cc:260] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.491705 17397 raft_consensus.cc:348] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.491926 17397 raft_consensus.cc:374] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.492043 17397 raft_consensus.cc:729] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.492337 17397 consensus_queue.cc:260] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.492933 17397 ts_tablet_manager.cc:1428] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:26.494029 17398 ts_tablet_manager.cc:1428] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:03:26.525676 17407 raft_consensus.cc:482] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:26.525895 17407 raft_consensus.cc:504] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.526626 17407 leader_election.cc:288] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255), 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:26.527170 17322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee938f803846463ab68e78530384f4b8" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" is_pre_election: true
I20260928 15:03:26.527199 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee938f803846463ab68e78530384f4b8" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892" is_pre_election: true
I20260928 15:03:26.527417 17322 raft_consensus.cc:2454] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 0.
I20260928 15:03:26.527464 17049 raft_consensus.cc:2454] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 0.
I20260928 15:03:26.527869 17115 leader_election.cc:302] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:26.528148 17407 raft_consensus.cc:2781] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:26.528290 17407 raft_consensus.cc:482] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:26.528429 17407 raft_consensus.cc:3037] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.530678 17407 raft_consensus.cc:504] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.531275 17407 leader_election.cc:288] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255), 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:26.531759 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee938f803846463ab68e78530384f4b8" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892"
I20260928 15:03:26.531750 17322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee938f803846463ab68e78530384f4b8" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
I20260928 15:03:26.531934 17322 raft_consensus.cc:3037] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.531937 17049 raft_consensus.cc:3037] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.533943 17049 raft_consensus.cc:2454] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 1.
I20260928 15:03:26.534323 17114 leader_election.cc:302] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe519763ff741a7b3b60e50233f5892, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:26.534585 17407 raft_consensus.cc:2781] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:26.534739 17407 raft_consensus.cc:686] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 1 LEADER]: Becoming Leader. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Running, Role: LEADER
I20260928 15:03:26.534825 17322 raft_consensus.cc:2454] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 1.
I20260928 15:03:26.534988 17407 consensus_queue.cc:237] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.537945 16885 catalog_manager.cc:5495] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to d42fd6f340854d89b0d6d82683302b00 (127.9.12.194). New cstate: current_term: 1 leader_uuid: "d42fd6f340854d89b0d6d82683302b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607804, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.254@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:26.589766 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.574460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39749 (local address 127.0.0.1:38290)
0928 15:03:26.575213 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.575236 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.575385 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.575806 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.575809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.576577 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:26.576867 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.576876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.578490 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.578494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.579540 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.579552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.580529 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.580534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.580661 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.581352 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:26.581388 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:26.583760 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:26.587455 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:26.587464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:26.587470 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:26.587866 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:26.588247 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:26.588250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:26.588252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:26.588425 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:26.589023 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:26.589029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:26.589172 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.589300 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.589481 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":356,"threads_started":1}
I20260928 15:03:26.590271 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.574517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.9.12.254:39749)
0928 15:03:26.575482 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.575488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.575512 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:03:26.575575 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.575581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.575661 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.575775 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.577049 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.578367 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.579667 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.580357 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.580485 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.583974 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:26.583981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:26.583986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:26.584022 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:26.587333 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:26.587976 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:26.587981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:26.587985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:26.588145 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:26.588527 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:26.588532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:26.588536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:26.588899 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:26.589051 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.589410 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.589652 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":628,"threads_started":1}
I20260928 15:03:26.595181 16885 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:38290:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:03:26.596385 16885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:26.603078 17024 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0c064595d54c63833beaf1b8890cc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ee0860f17ec4d279f259fc758fcd64d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:03:26.603237 17163 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0c064595d54c63833beaf1b8890cc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ee0860f17ec4d279f259fc758fcd64d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:03:26.603677 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0c064595d54c63833beaf1b8890cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.603747 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0c064595d54c63833beaf1b8890cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.603744 17302 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0c064595d54c63833beaf1b8890cc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ee0860f17ec4d279f259fc758fcd64d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:03:26.604308 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0c064595d54c63833beaf1b8890cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:26.611370 17397 tablet_bootstrap.cc:492] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: Bootstrap starting.
I20260928 15:03:26.612278 17398 tablet_bootstrap.cc:492] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: Bootstrap starting.
I20260928 15:03:26.612892 17397 tablet_bootstrap.cc:654] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.613842 17398 tablet_bootstrap.cc:654] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.614077 17396 tablet_bootstrap.cc:492] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: Bootstrap starting.
I20260928 15:03:26.615815 17396 tablet_bootstrap.cc:654] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:26.615957 17398 tablet_bootstrap.cc:492] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: No bootstrap required, opened a new log
I20260928 15:03:26.616115 17398 ts_tablet_manager.cc:1397] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 15:03:26.616883 17398 raft_consensus.cc:348] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.617152 17398 raft_consensus.cc:374] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.617231 17398 raft_consensus.cc:729] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca6626cc1f54bb78745432a42d2fbdb, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.617331 17397 tablet_bootstrap.cc:492] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: No bootstrap required, opened a new log
I20260928 15:03:26.617499 17397 ts_tablet_manager.cc:1397] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 15:03:26.617841 17398 consensus_queue.cc:260] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.618273 17397 raft_consensus.cc:348] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.618420 17398 ts_tablet_manager.cc:1428] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:26.618530 17397 raft_consensus.cc:374] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.618345 17404 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.618607 17397 raft_consensus.cc:729] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.618654 17404 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.618866 17397 consensus_queue.cc:260] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.619418 17397 ts_tablet_manager.cc:1428] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 15:03:26.619338 17407 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.619635 17407 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.621076 17396 tablet_bootstrap.cc:492] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: No bootstrap required, opened a new log
I20260928 15:03:26.621232 17396 ts_tablet_manager.cc:1397] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 15:03:26.621942 17396 raft_consensus.cc:348] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.622189 17396 raft_consensus.cc:374] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:26.622269 17396 raft_consensus.cc:729] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Initialized, Role: FOLLOWER
I20260928 15:03:26.622862 17396 consensus_queue.cc:260] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.623513 17396 ts_tablet_manager.cc:1428] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:26.623400 17403 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.623780 17403 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.670051 17403 raft_consensus.cc:482] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:26.670277 17403 raft_consensus.cc:504] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.670991 17404 raft_consensus.cc:482] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:26.671165 17404 raft_consensus.cc:504] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.671887 17403 leader_election.cc:288] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723), 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:26.672590 17404 leader_election.cc:288] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255), d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723)
I20260928 15:03:26.691862 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.672752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:33525 (local address 127.9.12.194:35723)
0928 15:03:26.673262 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.673269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.684065 (+ 10796us) server_negotiation.cc:407] Connection header received
0928 15:03:26.684375 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.684384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.684460 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.684530 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.685385 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.686705 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.689308 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.690599 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.690731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.690810 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.691422 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.691555 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260928 15:03:26.692550 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.672210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:35723 (local address 127.9.12.193:33525)
0928 15:03:26.683967 (+ 11757us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.683993 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.684177 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.684740 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.684744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.684811 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.685207 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.685220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.686875 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.686903 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.689111 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.689126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.691025 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.691031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.691228 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.691275 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.692047 (+ 772us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.692169 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11633,"mutex_wait_us":3,"spinlock_wait_cycles":2688,"thread_start_us":11509,"threads_started":1}
I20260928 15:03:26.692625 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.673109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:54633 (local address 127.9.12.193:38255)
0928 15:03:26.673314 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.673322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.684641 (+ 11319us) server_negotiation.cc:407] Connection header received
0928 15:03:26.684910 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.684916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.684993 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.685076 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.686176 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.687280 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.689883 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.691209 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.691330 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.691408 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.691898 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.692027 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:03:26.693589 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.672746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38255 (local address 127.9.12.195:54633)
0928 15:03:26.684555 (+ 11809us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.684576 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.684734 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.685271 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.685277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.685307 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.685590 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.685635 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.687496 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.687501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.689703 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.689720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.691548 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.691553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.691721 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.691750 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.693054 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.693205 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11618,"thread_start_us":10571,"threads_started":1}
I20260928 15:03:26.693621 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fd6f340854d89b0d6d82683302b00" is_pre_election: true
I20260928 15:03:26.694147 17183 raft_consensus.cc:2454] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe519763ff741a7b3b60e50233f5892 in term 0.
I20260928 15:03:26.695107 16975 leader_election.cc:302] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe519763ff741a7b3b60e50233f5892, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:26.695525 17403 raft_consensus.cc:2781] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:26.695650 17403 raft_consensus.cc:482] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:26.695750 17403 raft_consensus.cc:3037] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.695781 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892" is_pre_election: true
I20260928 15:03:26.697378 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.673436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:36905 (local address 127.9.12.194:35723)
0928 15:03:26.682667 (+ 9231us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.682675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.687139 (+ 4464us) server_negotiation.cc:407] Connection header received
0928 15:03:26.688158 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.688162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.688235 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.688334 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.690438 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.691759 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.694267 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.695739 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.695902 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.696007 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.696942 (+ 935us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.697070 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9063,"thread_start_us":8884,"threads_started":1}
I20260928 15:03:26.698035 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.673021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:35723 (local address 127.9.12.195:36905)
0928 15:03:26.687039 (+ 14018us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.687061 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.687974 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.688657 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.688663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.688692 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.689056 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.689067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.691949 (+ 2882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.691954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.694017 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.694033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.696553 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.696559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.696762 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.696794 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.697595 (+ 801us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.697719 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13845,"thread_start_us":14125,"threads_started":1}
I20260928 15:03:26.698565 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fd6f340854d89b0d6d82683302b00" is_pre_election: true
I20260928 15:03:26.698786 17183 raft_consensus.cc:2454] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca6626cc1f54bb78745432a42d2fbdb in term 0.
I20260928 15:03:26.698931 17403 raft_consensus.cc:504] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.699262 17254 leader_election.cc:302] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:26.699426 17049 raft_consensus.cc:2379] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ca6626cc1f54bb78745432a42d2fbdb in current term 1: Already voted for candidate 9fe519763ff741a7b3b60e50233f5892 in this term.
I20260928 15:03:26.699666 17404 raft_consensus.cc:2781] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:26.699774 17404 raft_consensus.cc:482] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:26.699859 17404 raft_consensus.cc:3037] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.701150 17403 leader_election.cc:288] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 election: Requested vote from peers d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723), 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:26.701715 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fd6f340854d89b0d6d82683302b00"
I20260928 15:03:26.701947 17183 raft_consensus.cc:3037] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:26.702793 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.684315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42097 (local address 127.9.12.195:45265)
0928 15:03:26.684487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.684496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.687840 (+ 3344us) server_negotiation.cc:407] Connection header received
0928 15:03:26.689439 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.689444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.689531 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.689593 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:26.694230 (+ 4637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.695246 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.700179 (+ 4933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.701450 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.701579 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.701656 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.702433 (+ 777us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.702571 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:03:26.702873 17404 raft_consensus.cc:504] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.703547 17404 leader_election.cc:288] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [CANDIDATE]: Term 1 election: Requested vote from peers 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255), d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723)
I20260928 15:03:26.703740 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.684152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:45265 (local address 127.9.12.193:42097)
0928 15:03:26.687751 (+ 3599us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.687768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.689298 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.689768 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.689772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.689797 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:26.690248 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.690262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.697364 (+ 7102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.697369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.700026 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.700040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.701954 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.701964 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.702245 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.702288 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.703146 (+ 858us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.703300 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3469,"thread_start_us":4161,"threads_started":1}
I20260928 15:03:26.704228 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fd6f340854d89b0d6d82683302b00"
I20260928 15:03:26.704742 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892"
I20260928 15:03:26.705032 17049 raft_consensus.cc:2379] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ca6626cc1f54bb78745432a42d2fbdb in current term 1: Already voted for candidate 9fe519763ff741a7b3b60e50233f5892 in this term.
I20260928 15:03:26.705192 17322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" is_pre_election: true
I20260928 15:03:26.705579 17322 raft_consensus.cc:2379] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fe519763ff741a7b3b60e50233f5892 in current term 1: Already voted for candidate 8ca6626cc1f54bb78745432a42d2fbdb in this term.
I20260928 15:03:26.705783 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c064595d54c63833beaf1b8890cc7" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
I20260928 15:03:26.706218 17321 raft_consensus.cc:2379] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fe519763ff741a7b3b60e50233f5892 in current term 1: Already voted for candidate 8ca6626cc1f54bb78745432a42d2fbdb in this term.
I20260928 15:03:26.706400 17183 raft_consensus.cc:2454] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe519763ff741a7b3b60e50233f5892 in term 1.
I20260928 15:03:26.705968 17253 leader_election.cc:302] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb; no voters: 9fe519763ff741a7b3b60e50233f5892, d42fd6f340854d89b0d6d82683302b00
I20260928 15:03:26.706789 16975 leader_election.cc:302] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fe519763ff741a7b3b60e50233f5892, d42fd6f340854d89b0d6d82683302b00; no voters: 8ca6626cc1f54bb78745432a42d2fbdb
I20260928 15:03:26.706928 17404 raft_consensus.cc:2726] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 15:03:26.707113 17403 raft_consensus.cc:2781] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:26.707619 17403 raft_consensus.cc:686] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Running, Role: LEADER
I20260928 15:03:26.708017 17403 consensus_queue.cc:237] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:26.710201 17441 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: TxnStatusTablet state changed. Reason: New leader 9fe519763ff741a7b3b60e50233f5892. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.710562 17441 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:03:26.711218 17443 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:03:26.711436 17443 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:03:26.711517 16886 catalog_manager.cc:5495] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193). New cstate: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:26.731074 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.724062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.9.12.193:38255)
0928 15:03:26.724231 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.724238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.724256 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:26.724561 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.724567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.724666 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.724801 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:26.725240 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.726497 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.728440 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.729150 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.729289 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.729777 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.730446 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.730586 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.730702 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.730794 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:03:26.731441 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.723963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38255 (local address 127.0.0.1:40784)
0928 15:03:26.724190 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.724209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.724366 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.724842 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.724847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.724883 (+ 36us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:03:26.725074 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.725087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.726664 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.726669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.728249 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.728263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.729343 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.729350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.729510 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.729578 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.730560 (+ 982us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.730581 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.730666 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.730810 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:03:26.736243 17321 raft_consensus.cc:1261] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Refusing update from remote peer 9fe519763ff741a7b3b60e50233f5892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:26.736394 17183 raft_consensus.cc:1261] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Refusing update from remote peer 9fe519763ff741a7b3b60e50233f5892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:26.737186 17441 consensus_queue.cc:1035] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:26.737820 17441 consensus_queue.cc:1035] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:26.751559 17407 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: TxnStatusTablet state changed. Reason: New leader 9fe519763ff741a7b3b60e50233f5892. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.751883 17407 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.754230 17403 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.754560 17403 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:03:26.759001 17407 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.759287 17407 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P d42fd6f340854d89b0d6d82683302b00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.761955 17404 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: TxnStatusTablet state changed. Reason: New leader 9fe519763ff741a7b3b60e50233f5892. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.762277 17404 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.762380 17441 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.762620 17441 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 9fe519763ff741a7b3b60e50233f5892: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:03:26.765618 17404 tablet_replica.cc:439] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } }
I20260928 15:03:26.765887 17404 tablet_replica.cc:442] T 9a0c064595d54c63833beaf1b8890cc7 P 8ca6626cc1f54bb78745432a42d2fbdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:26.785104 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.777374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38255 (local address 127.9.12.194:54407)
0928 15:03:26.778035 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.778060 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.778214 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.778611 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.778615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.778628 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:03:26.779019 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.779031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.780533 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.780539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.782448 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.782462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.783572 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.783579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.783811 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.783875 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.784451 (+ 576us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.784467 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.784566 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.784702 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":334,"threads_started":1}
I20260928 15:03:26.785097 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.777241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:54407 (local address 127.9.12.193:38255)
0928 15:03:26.777457 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.777463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.778172 (+ 709us) server_negotiation.cc:407] Connection header received
0928 15:03:26.778361 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.778366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.778457 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.778561 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:26.779202 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.780376 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.782609 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.783375 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.783511 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.784084 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.784255 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.784412 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.784606 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.784686 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 15:03:26.796820 17321 raft_consensus.cc:1261] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Refusing update from remote peer d42fd6f340854d89b0d6d82683302b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:26.796936 17049 raft_consensus.cc:1261] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Refusing update from remote peer d42fd6f340854d89b0d6d82683302b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:26.797514 17402 consensus_queue.cc:1035] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:26.797981 17402 consensus_queue.cc:1035] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:26.814129 17410 mvcc.cc:205] Tried to move back new op lower bound from 7334329576633610240 to 7334329575569891328. Current Snapshot: MvccSnapshot[applied={T|T < 7334329576633610240}]
I20260928 15:03:26.867887 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.850016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:55089 (local address 127.9.12.194:35723)
0928 15:03:26.850198 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.850206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.861132 (+ 10926us) server_negotiation.cc:407] Connection header received
0928 15:03:26.861558 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.861564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.861661 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.861743 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:26.863115 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.864231 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.865827 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.866277 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.866415 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.866917 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.867032 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.867133 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:26.867500 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 15:03:26.867585 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:03:26.868954 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.850050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:35723 (local address 127.9.12.193:55089)
0928 15:03:26.860992 (+ 10942us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.861012 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.861317 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.862282 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.862286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.862300 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:03:26.862709 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.862721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.864424 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.864434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.865670 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.865686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.866474 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.866481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.866639 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.866707 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:03:26.867319 (+ 612us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:03:26.867331 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:03:26.868419 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0928 15:03:26.868551 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10504,"thread_start_us":11014,"threads_started":1}
I20260928 15:03:26.918537 16885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:26.918794 16885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:26.921957 16885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Sending DeleteTablet for 3 replicas of tablet ee938f803846463ab68e78530384f4b8
I20260928 15:03:26.923054 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet ee938f803846463ab68e78530384f4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:26 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48722
I20260928 15:03:26.923044 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet ee938f803846463ab68e78530384f4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:26 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:33844
I20260928 15:03:26.923312 17420 tablet_replica.cc:330] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: stopping tablet replica
I20260928 15:03:26.923324 17419 tablet_replica.cc:330] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: stopping tablet replica
I20260928 15:03:26.923503 17420 raft_consensus.cc:2229] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:26.923537 17419 raft_consensus.cc:2229] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:26.923704 17419 raft_consensus.cc:2258] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:26.923833 17420 raft_consensus.cc:2258] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:26.924118 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet ee938f803846463ab68e78530384f4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:26 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:40780
I20260928 15:03:26.924993 17421 tablet_replica.cc:330] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: stopping tablet replica
I20260928 15:03:26.925232 17421 raft_consensus.cc:2229] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:26.925407 17421 raft_consensus.cc:2258] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:26.925401 17419 ts_tablet_manager.cc:1905] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:26.925607 17420 ts_tablet_manager.cc:1905] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:26.929389 17421 ts_tablet_manager.cc:1905] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:26.933321 17419 ts_tablet_manager.cc:1918] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:26.933527 17419 log.cc:1196] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/ee938f803846463ab68e78530384f4b8
I20260928 15:03:26.934954 17419 ts_tablet_manager.cc:1939] T ee938f803846463ab68e78530384f4b8 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting consensus metadata
I20260928 15:03:26.937260 16872 catalog_manager.cc:4841] TS 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265): tablet ee938f803846463ab68e78530384f4b8 (table test-table [id=3bd952f0e92f446fa6b105d5a2ebe01e]) successfully deleted
I20260928 15:03:26.940845 17420 ts_tablet_manager.cc:1918] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:26.941010 17420 log.cc:1196] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/ee938f803846463ab68e78530384f4b8
I20260928 15:03:26.941510 17420 ts_tablet_manager.cc:1939] T ee938f803846463ab68e78530384f4b8 P d42fd6f340854d89b0d6d82683302b00: Deleting consensus metadata
I20260928 15:03:26.942523 17421 ts_tablet_manager.cc:1918] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:26.942689 17421 log.cc:1196] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/ee938f803846463ab68e78530384f4b8
I20260928 15:03:26.942675 16872 catalog_manager.cc:4841] TS d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723): tablet ee938f803846463ab68e78530384f4b8 (table test-table [id=3bd952f0e92f446fa6b105d5a2ebe01e]) successfully deleted
I20260928 15:03:26.943114 17421 ts_tablet_manager.cc:1939] T ee938f803846463ab68e78530384f4b8 P 9fe519763ff741a7b3b60e50233f5892: Deleting consensus metadata
I20260928 15:03:26.944229 16871 catalog_manager.cc:4841] TS 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255): tablet ee938f803846463ab68e78530384f4b8 (table test-table [id=3bd952f0e92f446fa6b105d5a2ebe01e]) successfully deleted
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:26 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:27.003305 17432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:26.989858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.9.12.254:39749)
0928 15:03:26.990084 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.990092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.990663 (+ 571us) server_negotiation.cc:407] Connection header received
0928 15:03:26.990849 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.990856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.990958 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.991071 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.992769 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.993889 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.995745 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.996394 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.996508 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.003050 (+ 6542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38298: BlockingRecv error: recv got EOF from 127.0.0.1:38298 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 15:03:27 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM
I20260928 15:03:27.066128 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.036112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.9.12.254:39749)
0928 15:03:27.036312 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.036318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.048876 (+ 12558us) server_negotiation.cc:407] Connection header received
0928 15:03:27.049128 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.049135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.049225 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.049303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.051260 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.052373 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.054120 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.054722 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.054812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.060043 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:27.060051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:27.060056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:27.060100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:27.063429 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.064034 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.064039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.064043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.064191 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.064614 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.064619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.064622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.064996 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:27.065254 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.065491 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.065769 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:03:27.069804 16885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:27.070492 16885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:27.077921 17302 tablet_service.cc:1467] Processing CreateTablet for tablet c22423fa38114be6b0a8e52cf393cf20 (DEFAULT_TABLE table=test-table [id=6189349f8e864f14a335d66c8341d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:27.078192 17163 tablet_service.cc:1467] Processing CreateTablet for tablet c22423fa38114be6b0a8e52cf393cf20 (DEFAULT_TABLE table=test-table [id=6189349f8e864f14a335d66c8341d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:27.078222 17024 tablet_service.cc:1467] Processing CreateTablet for tablet c22423fa38114be6b0a8e52cf393cf20 (DEFAULT_TABLE table=test-table [id=6189349f8e864f14a335d66c8341d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:27.078382 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22423fa38114be6b0a8e52cf393cf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:27.078605 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22423fa38114be6b0a8e52cf393cf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:27.078613 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22423fa38114be6b0a8e52cf393cf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:27.085008 17398 tablet_bootstrap.cc:492] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Bootstrap starting.
I20260928 15:03:27.086301 17396 tablet_bootstrap.cc:492] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Bootstrap starting.
I20260928 15:03:27.086714 17398 tablet_bootstrap.cc:654] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:27.088060 17396 tablet_bootstrap.cc:654] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:27.089588 17397 tablet_bootstrap.cc:492] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Bootstrap starting.
I20260928 15:03:27.090854 17397 tablet_bootstrap.cc:654] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:27.092666 17396 tablet_bootstrap.cc:492] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: No bootstrap required, opened a new log
I20260928 15:03:27.092841 17396 ts_tablet_manager.cc:1397] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260928 15:03:27.093596 17396 raft_consensus.cc:348] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.093818 17396 raft_consensus.cc:374] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:27.093946 17396 raft_consensus.cc:729] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Initialized, Role: FOLLOWER
I20260928 15:03:27.094220 17396 consensus_queue.cc:260] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.094730 17396 ts_tablet_manager.cc:1428] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:03:27.095881 17397 tablet_bootstrap.cc:492] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: No bootstrap required, opened a new log
I20260928 15:03:27.096035 17397 ts_tablet_manager.cc:1397] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260928 15:03:27.096822 17397 raft_consensus.cc:348] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.097078 17397 raft_consensus.cc:374] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:27.097167 17397 raft_consensus.cc:729] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Initialized, Role: FOLLOWER
I20260928 15:03:27.097400 17397 consensus_queue.cc:260] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.097951 17397 ts_tablet_manager.cc:1428] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:27.101501 17398 tablet_bootstrap.cc:492] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: No bootstrap required, opened a new log
I20260928 15:03:27.101663 17398 ts_tablet_manager.cc:1397] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20260928 15:03:27.102408 17398 raft_consensus.cc:348] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.102674 17398 raft_consensus.cc:374] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:27.102783 17398 raft_consensus.cc:729] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca6626cc1f54bb78745432a42d2fbdb, State: Initialized, Role: FOLLOWER
I20260928 15:03:27.103020 17398 consensus_queue.cc:260] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.103644 17398 ts_tablet_manager.cc:1428] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:27.167078 17456 raft_consensus.cc:482] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:27.167299 17456 raft_consensus.cc:504] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.168125 17456 leader_election.cc:288] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255), 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:27.168671 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22423fa38114be6b0a8e52cf393cf20" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" is_pre_election: true
I20260928 15:03:27.168725 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22423fa38114be6b0a8e52cf393cf20" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892" is_pre_election: true
I20260928 15:03:27.168936 17321 raft_consensus.cc:2454] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 0.
I20260928 15:03:27.169008 17049 raft_consensus.cc:2454] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 0.
I20260928 15:03:27.169417 17115 leader_election.cc:302] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:27.169725 17456 raft_consensus.cc:2781] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:27.169871 17456 raft_consensus.cc:482] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:27.169966 17456 raft_consensus.cc:3037] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:27.172111 17456 raft_consensus.cc:504] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.172684 17456 leader_election.cc:288] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255), 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265)
I20260928 15:03:27.173183 17049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22423fa38114be6b0a8e52cf393cf20" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe519763ff741a7b3b60e50233f5892"
I20260928 15:03:27.173306 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22423fa38114be6b0a8e52cf393cf20" candidate_uuid: "d42fd6f340854d89b0d6d82683302b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
I20260928 15:03:27.173427 17049 raft_consensus.cc:3037] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:27.173506 17321 raft_consensus.cc:3037] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:27.176365 17321 raft_consensus.cc:2454] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 1.
I20260928 15:03:27.176630 17049 raft_consensus.cc:2454] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42fd6f340854d89b0d6d82683302b00 in term 1.
I20260928 15:03:27.176791 17115 leader_election.cc:302] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:27.177062 17456 raft_consensus.cc:2781] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:27.177238 17456 raft_consensus.cc:686] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 1 LEADER]: Becoming Leader. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Running, Role: LEADER
I20260928 15:03:27.177520 17456 consensus_queue.cc:237] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } }
I20260928 15:03:27.180358 16885 catalog_manager.cc:5495] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to d42fd6f340854d89b0d6d82683302b00 (127.9.12.194). New cstate: current_term: 1 leader_uuid: "d42fd6f340854d89b0d6d82683302b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:27.215826 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.210523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.9.12.194:35723)
0928 15:03:27.210715 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.210720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.210822 (+ 102us) server_negotiation.cc:407] Connection header received
0928 15:03:27.211019 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.211022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.211086 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.211152 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:27.211542 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.212253 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.214212 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.214587 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.214681 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.215121 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:27.215219 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:27.215280 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.215466 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.215562 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:03:27.218617 17049 raft_consensus.cc:1261] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Refusing update from remote peer d42fd6f340854d89b0d6d82683302b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:27.218613 17321 raft_consensus.cc:1261] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Refusing update from remote peer d42fd6f340854d89b0d6d82683302b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:27.219235 17407 consensus_queue.cc:1035] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:27.219745 17456 consensus_queue.cc:1035] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:27.233471 16887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:27.233703 16887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:27.237406 16887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Sending DeleteTablet for 3 replicas of tablet c22423fa38114be6b0a8e52cf393cf20
I20260928 15:03:27.238085 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet c22423fa38114be6b0a8e52cf393cf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:40780
I20260928 15:03:27.238377 17421 tablet_replica.cc:330] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: stopping tablet replica
I20260928 15:03:27.238361 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet c22423fa38114be6b0a8e52cf393cf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48722
I20260928 15:03:27.238570 17421 raft_consensus.cc:2229] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:27.238570 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet c22423fa38114be6b0a8e52cf393cf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:33844
I20260928 15:03:27.238790 17421 raft_consensus.cc:2258] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:27.238826 17301 tablet_service.cc:1514] Processing DeleteTablet for tablet c22423fa38114be6b0a8e52cf393cf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:33844
I20260928 15:03:27.239053 17419 tablet_replica.cc:330] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: stopping tablet replica
I20260928 15:03:27.239068 17420 tablet_replica.cc:330] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: stopping tablet replica
I20260928 15:03:27.239229 17420 raft_consensus.cc:2229] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:27.239229 17419 raft_consensus.cc:2229] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:27.239434 17419 raft_consensus.cc:2258] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:27.239538 17420 raft_consensus.cc:2258] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:27.239573 16885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:27.240341 16885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:27.240720 17419 ts_tablet_manager.cc:1905] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:27.240792 17421 ts_tablet_manager.cc:1905] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:27.242911 17420 ts_tablet_manager.cc:1905] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:27.246172 17421 ts_tablet_manager.cc:1918] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:27.246311 17421 log.cc:1196] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/c22423fa38114be6b0a8e52cf393cf20
I20260928 15:03:27.246711 17421 ts_tablet_manager.cc:1939] T c22423fa38114be6b0a8e52cf393cf20 P 9fe519763ff741a7b3b60e50233f5892: Deleting consensus metadata
I20260928 15:03:27.247174 17419 ts_tablet_manager.cc:1918] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:27.247298 17419 log.cc:1196] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/c22423fa38114be6b0a8e52cf393cf20
I20260928 15:03:27.247696 17419 ts_tablet_manager.cc:1939] T c22423fa38114be6b0a8e52cf393cf20 P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting consensus metadata
I20260928 15:03:27.247682 16871 catalog_manager.cc:4841] TS 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255): tablet c22423fa38114be6b0a8e52cf393cf20 (table test-table [id=6189349f8e864f14a335d66c8341d627]) successfully deleted
I20260928 15:03:27.248741 16872 catalog_manager.cc:4841] TS 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265): tablet c22423fa38114be6b0a8e52cf393cf20 (table test-table [id=6189349f8e864f14a335d66c8341d627]) successfully deleted
I20260928 15:03:27.249279 17420 ts_tablet_manager.cc:1918] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:27.249411 17420 log.cc:1196] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/c22423fa38114be6b0a8e52cf393cf20
I20260928 15:03:27.249836 17420 ts_tablet_manager.cc:1939] T c22423fa38114be6b0a8e52cf393cf20 P d42fd6f340854d89b0d6d82683302b00: Deleting consensus metadata
I20260928 15:03:27.250840 16872 catalog_manager.cc:4841] TS d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723): tablet c22423fa38114be6b0a8e52cf393cf20 (table test-table [id=6189349f8e864f14a335d66c8341d627]) successfully deleted
W20260928 15:03:27.251253 16872 catalog_manager.cc:4808] TS 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265): delete failed for tablet c22423fa38114be6b0a8e52cf393cf20 because the tablet was not found. No further retry: Not found: Tablet not found: c22423fa38114be6b0a8e52cf393cf20
I20260928 15:03:27.252509 17301 tablet_service.cc:1467] Processing CreateTablet for tablet 917095a26f474b3e878253c9244228ee (DEFAULT_TABLE table=test-table [id=86e73848eeff4c438d3a1adfa7e1f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:27.252846 17163 tablet_service.cc:1467] Processing CreateTablet for tablet 917095a26f474b3e878253c9244228ee (DEFAULT_TABLE table=test-table [id=86e73848eeff4c438d3a1adfa7e1f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:27.252935 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917095a26f474b3e878253c9244228ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:27.253145 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917095a26f474b3e878253c9244228ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:27.256644 17024 tablet_service.cc:1467] Processing CreateTablet for tablet 917095a26f474b3e878253c9244228ee (DEFAULT_TABLE table=test-table [id=86e73848eeff4c438d3a1adfa7e1f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:27.257121 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917095a26f474b3e878253c9244228ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:27.257606 17398 tablet_bootstrap.cc:492] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Bootstrap starting.
I20260928 15:03:27.257964 17397 tablet_bootstrap.cc:492] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Bootstrap starting.
I20260928 15:03:27.259114 17397 tablet_bootstrap.cc:654] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:27.260818 17397 tablet_bootstrap.cc:492] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: No bootstrap required, opened a new log
I20260928 15:03:27.260987 17397 ts_tablet_manager.cc:1397] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:27.261746 17396 tablet_bootstrap.cc:492] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Bootstrap starting.
I20260928 15:03:27.261605 17397 raft_consensus.cc:348] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.261857 17397 raft_consensus.cc:374] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:27.261933 17397 raft_consensus.cc:729] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42fd6f340854d89b0d6d82683302b00, State: Initialized, Role: FOLLOWER
I20260928 15:03:27.262168 17397 consensus_queue.cc:260] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.262838 17397 ts_tablet_manager.cc:1428] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:27.263326 17396 tablet_bootstrap.cc:654] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:27.264997 17398 tablet_bootstrap.cc:654] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:27.265319 17396 tablet_bootstrap.cc:492] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: No bootstrap required, opened a new log
I20260928 15:03:27.265516 17396 ts_tablet_manager.cc:1397] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:03:27.266287 17396 raft_consensus.cc:348] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.266523 17396 raft_consensus.cc:374] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:27.266602 17396 raft_consensus.cc:729] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Initialized, Role: FOLLOWER
I20260928 15:03:27.266865 17396 consensus_queue.cc:260] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.267395 17398 tablet_bootstrap.cc:492] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: No bootstrap required, opened a new log
I20260928 15:03:27.267526 17396 ts_tablet_manager.cc:1428] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:03:27.267552 17398 ts_tablet_manager.cc:1397] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260928 15:03:27.268350 17398 raft_consensus.cc:348] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.268584 17398 raft_consensus.cc:374] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:27.268668 17398 raft_consensus.cc:729] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca6626cc1f54bb78745432a42d2fbdb, State: Initialized, Role: FOLLOWER
I20260928 15:03:27.268926 17398 consensus_queue.cc:260] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.269431 17398 ts_tablet_manager.cc:1428] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:27.340206 17403 raft_consensus.cc:482] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:27.340453 17403 raft_consensus.cc:504] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.341171 17403 leader_election.cc:288] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265), d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723)
I20260928 15:03:27.341647 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917095a26f474b3e878253c9244228ee" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" is_pre_election: true
I20260928 15:03:27.341748 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917095a26f474b3e878253c9244228ee" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fd6f340854d89b0d6d82683302b00" is_pre_election: true
I20260928 15:03:27.341897 17321 raft_consensus.cc:2454] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe519763ff741a7b3b60e50233f5892 in term 0.
I20260928 15:03:27.342053 17182 raft_consensus.cc:2454] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe519763ff741a7b3b60e50233f5892 in term 0.
I20260928 15:03:27.342337 16975 leader_election.cc:302] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca6626cc1f54bb78745432a42d2fbdb, 9fe519763ff741a7b3b60e50233f5892; no voters:
I20260928 15:03:27.342644 17403 raft_consensus.cc:2781] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:27.342743 17403 raft_consensus.cc:482] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:27.342856 17403 raft_consensus.cc:3037] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:27.344892 17403 raft_consensus.cc:504] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.345527 17403 leader_election.cc:288] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 election: Requested vote from peers 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265), d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723)
I20260928 15:03:27.345956 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917095a26f474b3e878253c9244228ee" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6626cc1f54bb78745432a42d2fbdb"
I20260928 15:03:27.345988 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917095a26f474b3e878253c9244228ee" candidate_uuid: "9fe519763ff741a7b3b60e50233f5892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fd6f340854d89b0d6d82683302b00"
I20260928 15:03:27.346158 17321 raft_consensus.cc:3037] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:27.346184 17182 raft_consensus.cc:3037] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:27.348147 17182 raft_consensus.cc:2454] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe519763ff741a7b3b60e50233f5892 in term 1.
I20260928 15:03:27.348199 17321 raft_consensus.cc:2454] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe519763ff741a7b3b60e50233f5892 in term 1.
I20260928 15:03:27.348537 16975 leader_election.cc:302] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe519763ff741a7b3b60e50233f5892, d42fd6f340854d89b0d6d82683302b00; no voters:
I20260928 15:03:27.348773 17403 raft_consensus.cc:2781] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:27.348953 17403 raft_consensus.cc:686] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe519763ff741a7b3b60e50233f5892, State: Running, Role: LEADER
I20260928 15:03:27.349237 17403 consensus_queue.cc:237] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } }
I20260928 15:03:27.351657 16885 catalog_manager.cc:5495] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193). New cstate: current_term: 1 leader_uuid: "9fe519763ff741a7b3b60e50233f5892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe519763ff741a7b3b60e50233f5892" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:27.377017 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.371702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.9.12.193:38255)
0928 15:03:27.372149 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.372153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.372168 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:03:27.372214 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.372218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.372330 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.372421 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:27.372775 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.373490 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.374849 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.375573 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.375704 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.376199 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:27.376371 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:27.376457 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.376602 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.376677 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":262,"threads_started":1}
I20260928 15:03:27.389072 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.384527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:55189 (local address 127.9.12.193:38255)
0928 15:03:27.384686 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.384690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.385046 (+ 356us) server_negotiation.cc:407] Connection header received
0928 15:03:27.385173 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.385177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.385232 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.385299 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:27.385668 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.386362 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.387605 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.387988 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.388086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.388473 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:27.388564 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:27.388632 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.388758 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.388853 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":52}
I20260928 15:03:27.389190 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.384447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38255 (local address 127.9.12.193:55189)
0928 15:03:27.384961 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:27.384975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:27.385083 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:27.385322 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:27.385325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:27.385333 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:03:27.385565 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.385572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.386471 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.386474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.387494 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.387505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.388121 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.388124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.388249 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.388343 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:03:27.388646 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:03:27.388654 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:03:27.388723 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:03:27.388849 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":17,"thread_start_us":282,"threads_started":1}
I20260928 15:03:27.396461 17182 raft_consensus.cc:1261] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Refusing update from remote peer 9fe519763ff741a7b3b60e50233f5892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:27.396533 17321 raft_consensus.cc:1261] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Refusing update from remote peer 9fe519763ff741a7b3b60e50233f5892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:27.396982 17481 consensus_queue.cc:1035] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42fd6f340854d89b0d6d82683302b00" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:27.397316 17441 consensus_queue.cc:1035] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca6626cc1f54bb78745432a42d2fbdb" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:27.450489 16887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:27.450745 16887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:27.453704 16887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f8ef6cbbf14cbf84639ad8a6599161: Sending DeleteTablet for 3 replicas of tablet 917095a26f474b3e878253c9244228ee
I20260928 15:03:27.454506 17301 tablet_service.cc:1514] Processing DeleteTablet for tablet 917095a26f474b3e878253c9244228ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:33844
I20260928 15:03:27.454648 17023 tablet_service.cc:1514] Processing DeleteTablet for tablet 917095a26f474b3e878253c9244228ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:40780
I20260928 15:03:27.454823 17419 tablet_replica.cc:330] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: stopping tablet replica
I20260928 15:03:27.455029 17421 tablet_replica.cc:330] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: stopping tablet replica
I20260928 15:03:27.455070 17419 raft_consensus.cc:2229] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:27.455263 17419 raft_consensus.cc:2258] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:27.455272 17421 raft_consensus.cc:2229] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:27.455783 17421 raft_consensus.cc:2258] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:27.456458 17419 ts_tablet_manager.cc:1905] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:27.456842 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet 917095a26f474b3e878253c9244228ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:27 UTC) from {username='oryx', principal='oryx/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48722
I20260928 15:03:27.457150 17420 tablet_replica.cc:330] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: stopping tablet replica
I20260928 15:03:27.457350 17420 raft_consensus.cc:2229] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:27.457543 17420 raft_consensus.cc:2258] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:27.458616 17420 ts_tablet_manager.cc:1905] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:27.461078 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 16958
I20260928 15:03:27.461107 17421 ts_tablet_manager.cc:1905] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:27.466209 17420 ts_tablet_manager.cc:1918] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:27.466355 17420 log.cc:1196] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/917095a26f474b3e878253c9244228ee
I20260928 15:03:27.466760 17420 ts_tablet_manager.cc:1939] T 917095a26f474b3e878253c9244228ee P d42fd6f340854d89b0d6d82683302b00: Deleting consensus metadata
I20260928 15:03:27.467861 16872 catalog_manager.cc:4841] TS d42fd6f340854d89b0d6d82683302b00 (127.9.12.194:35723): tablet 917095a26f474b3e878253c9244228ee (table test-table [id=86e73848eeff4c438d3a1adfa7e1f7bc]) successfully deleted
I20260928 15:03:27.467993 17419 ts_tablet_manager.cc:1918] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:27.468135 17419 log.cc:1196] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/917095a26f474b3e878253c9244228ee
I20260928 15:03:27.468565 17419 ts_tablet_manager.cc:1939] T 917095a26f474b3e878253c9244228ee P 8ca6626cc1f54bb78745432a42d2fbdb: Deleting consensus metadata
I20260928 15:03:27.468778 17421 ts_tablet_manager.cc:1918] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:27.468928 17421 log.cc:1196] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/917095a26f474b3e878253c9244228ee
I20260928 15:03:27.469364 17421 ts_tablet_manager.cc:1939] T 917095a26f474b3e878253c9244228ee P 9fe519763ff741a7b3b60e50233f5892: Deleting consensus metadata
I20260928 15:03:27.469877 16872 catalog_manager.cc:4841] TS 8ca6626cc1f54bb78745432a42d2fbdb (127.9.12.195:45265): tablet 917095a26f474b3e878253c9244228ee (table test-table [id=86e73848eeff4c438d3a1adfa7e1f7bc]) successfully deleted
I20260928 15:03:27.470534 16871 catalog_manager.cc:4841] TS 9fe519763ff741a7b3b60e50233f5892 (127.9.12.193:38255): tablet 917095a26f474b3e878253c9244228ee (table test-table [id=86e73848eeff4c438d3a1adfa7e1f7bc]) successfully deleted
Sep 28 15:03:27 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.193@KRBTEST.COM
I20260928 15:03:27.486896 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.462754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.9.12.193:38255)
0928 15:03:27.462980 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.462987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.472727 (+ 9740us) server_negotiation.cc:407] Connection header received
0928 15:03:27.472934 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.472941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.473043 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.473145 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.474361 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.475292 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.476567 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.477019 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.477170 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.481095 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:27.481109 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:27.481124 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:27.481202 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:27.484136 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.484813 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.484823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.484826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.484974 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.485345 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.485350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.485356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.485728 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:27.486067 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.486292 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.486516 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:03:27.487607 17089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:27.617316 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 16958
I20260928 15:03:27.641124 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 17098
Sep 28 15:03:27 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.194@KRBTEST.COM
I20260928 15:03:27.655803 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.641944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.9.12.194:35723)
0928 15:03:27.642143 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.642147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.642182 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:03:27.642337 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.642340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.642407 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.642531 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.643723 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.644444 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.645709 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.646111 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.646203 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.650153 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:27.650166 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:27.650180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:27.650214 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:27.653090 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.653702 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.653709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.653711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.653888 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.654242 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.654247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.654249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.654572 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:27.654798 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.655230 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.655401 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:03:27.656585 17228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:27.779997 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 17098
I20260928 15:03:27.802227 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 17237
Sep 28 15:03:27 dist-test-slave-dmhs krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.12.195@KRBTEST.COM
I20260928 15:03:27.817476 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.803016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.9.12.195:45265)
0928 15:03:27.803439 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.803444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.803458 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:03:27.803507 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.803511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.803572 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.803654 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.805113 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.805816 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.807054 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.807475 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.807572 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.812027 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:27.812042 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:27.812058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:27.812097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:27.814759 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.815327 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.815333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.815336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.815470 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:27.815785 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:27.815788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:27.815790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:27.816109 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:27.816388 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:27.816946 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 15:03:27.817107 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":235,"threads_started":1}
I20260928 15:03:27.818116 17367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:27.911949 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 17237
I20260928 15:03:27.929041 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 16855
I20260928 15:03:27.930203 16927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:28.044416 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 16855
2026-09-28T15:03:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5330 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:28 dist-test-slave-dmhs krb5kdc[17499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:28 dist-test-slave-dmhs krb5kdc[17499](info): set up 2 sockets
Sep 28 15:03:28 dist-test-slave-dmhs krb5kdc[17499](info): commencing operation
krb5kdc: starting...
W20260928 15:03:29.121552 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:03:29 dist-test-slave-dmhs krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:03:29.332108 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45987
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:45987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:29.648548 17512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:29.648810 17512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:29.648906 17512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:29.655907 17512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:29.656008 17512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:29.656052 17512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:29.656101 17512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:29.656148 17512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:29.665503 17512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:45987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:29.666723 17512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:29.667991 17512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:29.679888 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:29.679912 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:29.679821 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:29.680646 17512 server_base.cc:1034] running on GCE node
I20260928 15:03:29.681337 17512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:29.682595 17512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:29.683813 17512 hybrid_clock.cc:648] HybridClock initialized: now 1790607809683768 us; error 57 us; skew 500 ppm
I20260928 15:03:29.684155 17512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T15:03:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1712 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:29 dist-test-slave-dmhs krb5kdc[17526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:29 dist-test-slave-dmhs krb5kdc[17526](info): set up 2 sockets
Sep 28 15:03:29 dist-test-slave-dmhs krb5kdc[17526](info): commencing operation
krb5kdc: starting...
W20260928 15:03:30.845535 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:03:30 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:03:31.059912 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33639
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:33639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:31.368697 17539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:31.369019 17539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:31.369184 17539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:31.376832 17539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:31.376962 17539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:31.377063 17539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:31.377169 17539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:31.377249 17539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:31.386967 17539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:33639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:31.388441 17539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:31.389962 17539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:31.402172 17544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:31.402366 17547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:31.402367 17545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:31.403285 17539 server_base.cc:1034] running on GCE node
I20260928 15:03:31.403971 17539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:31.405337 17539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:31.406541 17539 hybrid_clock.cc:648] HybridClock initialized: now 1790607811406505 us; error 32 us; skew 500 ppm
I20260928 15:03:31.406838 17539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:31 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607811, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:31.409783 17539 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:03:31.411132 17539 webserver.cc:460] Webserver started at http://127.9.12.254:40773/ using document root <none> and password file <none>
I20260928 15:03:31.411775 17539 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:31.411868 17539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:31.412106 17539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:31.414907 17539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "c1ae704b49cf48d3a38267234d355453"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "054aa51bf1b31acd0c20e785cb28492e"
server_key_iv: "dcab3dcb3a98903cb458286c13f9ddc1"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.415617 17539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "c1ae704b49cf48d3a38267234d355453"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "054aa51bf1b31acd0c20e785cb28492e"
server_key_iv: "dcab3dcb3a98903cb458286c13f9ddc1"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.420617 17539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:03:31.423641 17554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:31.424934 17539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:31.425113 17539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "c1ae704b49cf48d3a38267234d355453"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "054aa51bf1b31acd0c20e785cb28492e"
server_key_iv: "dcab3dcb3a98903cb458286c13f9ddc1"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.425326 17539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:31.440546 17539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:31.444042 17539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:31.444322 17539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:31.461967 17539 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:33639
I20260928 15:03:31.462004 17605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:33639 every 8 connection(s)
I20260928 15:03:31.463218 17539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:31.466751 17606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:31.469880 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 17539
I20260928 15:03:31.470088 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:31.470830 9267 external_mini_cluster.cc:1428] Setting key 2f608f31db9930e7260acdafe1026304
I20260928 15:03:31.478740 17606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Bootstrap starting.
I20260928 15:03:31.481722 17606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:31.482770 17606 log.cc:824] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:03:31 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:31.485499 17606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: No bootstrap required, opened a new log
I20260928 15:03:31.492545 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:31.472827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.9.12.254:33639)
0928 15:03:31.473489 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:31.473500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:31.473538 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:03:31.475502 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:31.475523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:31.475900 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:31.476320 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:31.477759 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.479119 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.480455 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.481128 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.481325 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.485501 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:31.485508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:31.485521 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:31.485553 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:31.488685 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:31.489376 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:31.489382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:31.489388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:31.489538 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:31.489930 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:31.489934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:31.489936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:31.490455 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:31.490804 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:31.491196 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 15:03:31.491590 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":283,"threads_started":1}
I20260928 15:03:31.493693 17606 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:31.494127 17606 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:31.494241 17606 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ae704b49cf48d3a38267234d355453, State: Initialized, Role: FOLLOWER
I20260928 15:03:31.494796 17606 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:31.495034 17606 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:31.495178 17606 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:31.495350 17606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:31.497880 17606 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:31.498287 17606 leader_election.cc:302] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1ae704b49cf48d3a38267234d355453; no voters:
I20260928 15:03:31.498787 17606 leader_election.cc:288] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:31.498904 17611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:31.499611 17611 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 LEADER]: Becoming Leader. State: Replica: c1ae704b49cf48d3a38267234d355453, State: Running, Role: LEADER
I20260928 15:03:31.500139 17611 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:31.500809 17606 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:31.502657 17612 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1ae704b49cf48d3a38267234d355453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } } }
I20260928 15:03:31.503037 17612 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:31.503587 17613 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ae704b49cf48d3a38267234d355453. Latest consensus state: current_term: 1 leader_uuid: "c1ae704b49cf48d3a38267234d355453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } } }
I20260928 15:03:31.503885 17613 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:31.505292 17618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:31.510708 17618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:31.519805 17618 catalog_manager.cc:1347] Generated new cluster ID: 37620e29ad564cd5b8ff99153fa436d4
I20260928 15:03:31.519933 17618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:31.548883 17618 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:31.550118 17618 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:31.563077 17618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Generated new TSK 0
I20260928 15:03:31.563885 17618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:03:31.623870 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33639
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:31.958561 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:31.958843 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:31.959007 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:31.966207 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:31.966320 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:31.966518 17632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:31.976356 17632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:31.977702 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:31.979012 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:31.992184 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:31.992244 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:31.992534 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:31.992980 17632 server_base.cc:1034] running on GCE node
I20260928 15:03:31.993655 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:31.994695 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:31.995968 17632 hybrid_clock.cc:648] HybridClock initialized: now 1790607811995947 us; error 55 us; skew 500 ppm
I20260928 15:03:31.996336 17632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:31 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607811, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:31.999593 17632 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:03:32.001343 17632 webserver.cc:460] Webserver started at http://127.9.12.193:37545/ using document root <none> and password file <none>
I20260928 15:03:32.002168 17632 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.002310 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.002619 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.005713 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "05ecae8485f743deabd847d1764adb19"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "49bacc7e0b11705e5ddf52ed6c31aa6a"
server_key_iv: "01033f29c869c42403b9e2ac20780ea5"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.006673 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "05ecae8485f743deabd847d1764adb19"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "49bacc7e0b11705e5ddf52ed6c31aa6a"
server_key_iv: "01033f29c869c42403b9e2ac20780ea5"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.012133 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:03:32.015646 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.017081 17632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:32.017268 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "05ecae8485f743deabd847d1764adb19"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "49bacc7e0b11705e5ddf52ed6c31aa6a"
server_key_iv: "01033f29c869c42403b9e2ac20780ea5"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.017496 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.039029 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.043514 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.043848 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.044898 17632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.046278 17632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.046382 17632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.046545 17632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.046622 17632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.080382 17632 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:43697
I20260928 15:03:32.080476 17759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:43697 every 8 connection(s)
I20260928 15:03:32.081743 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:32.087036 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 17632
I20260928 15:03:32.087247 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:32.088079 9267 external_mini_cluster.cc:1428] Setting key 6390e654213b5a7477f578c7461b8040
Sep 28 15:03:32 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607811, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:32.104372 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.084420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42785 (local address 127.9.12.254:33639)
0928 15:03:32.084852 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.084857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.085861 (+ 1004us) server_negotiation.cc:407] Connection header received
0928 15:03:32.088542 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.088547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.088615 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.088709 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.090536 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.091240 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.093777 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.094188 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.094287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.098444 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.098449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.098452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:32.098481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.101238 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:32.101999 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:32.102005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:32.102008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:32.102147 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:32.102606 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:32.102626 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:32.102629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:32.102878 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.103038 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.103730 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.103962 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260928 15:03:32.105329 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.084994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:42785)
0928 15:03:32.085696 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.085739 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.088185 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.088965 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.088973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.089630 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.090345 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.090369 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.091423 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.091431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.093630 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.093647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.094449 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.094460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.094787 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.095513 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:32.095554 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.098227 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.101366 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:32.101374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:32.101409 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:32.101830 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:32.102275 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:32.102280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:32.102284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:32.102483 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:32.103021 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.103029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.103350 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.103627 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.104144 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":269,"threads_started":1}
I20260928 15:03:32.107023 17760 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33639
I20260928 15:03:32.107368 17760 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.108340 17760 heartbeater.cc:511] Master 127.9.12.254:33639 requested a full tablet report, sending...
I20260928 15:03:32.110638 17571 ts_manager.cc:194] Registered new tserver with Master: 05ecae8485f743deabd847d1764adb19 (127.9.12.193:43697)
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:03:32.112125 17571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:42785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:03:32.139412 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33639
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:32.455649 17766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.455983 17766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.456135 17766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.463550 17766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.463698 17766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.463864 17766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:32.473563 17766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:32.474841 17766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.476078 17766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.489042 17771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.489104 17772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.489169 17774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.489959 17766 server_base.cc:1034] running on GCE node
I20260928 15:03:32.490516 17766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.491385 17766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.492617 17766 hybrid_clock.cc:648] HybridClock initialized: now 1790607812492585 us; error 44 us; skew 500 ppm
I20260928 15:03:32.492920 17766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:32 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:32.495924 17766 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:03:32.497259 17766 webserver.cc:460] Webserver started at http://127.9.12.194:42463/ using document root <none> and password file <none>
I20260928 15:03:32.497854 17766 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.497946 17766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.498185 17766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.500897 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "70fa94f109194edd8fc207024e115556"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-dmhs"
server_key: "02aaee7dbacab21f221cbcec4bf78e51"
server_key_iv: "0d352f1a8e081dc9f9a4b306a9bb8cf6"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.501751 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "70fa94f109194edd8fc207024e115556"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-dmhs"
server_key: "02aaee7dbacab21f221cbcec4bf78e51"
server_key_iv: "0d352f1a8e081dc9f9a4b306a9bb8cf6"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.506700 17766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:03:32.510138 17781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.511431 17766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:03:32.511601 17766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "70fa94f109194edd8fc207024e115556"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-dmhs"
server_key: "02aaee7dbacab21f221cbcec4bf78e51"
server_key_iv: "0d352f1a8e081dc9f9a4b306a9bb8cf6"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.511763 17766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.545506 17766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.549621 17766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.549914 17766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.551271 17766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.552673 17766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.552770 17766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.552894 17766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.552968 17766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.585493 17766 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:45433
I20260928 15:03:32.585574 17893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:45433 every 8 connection(s)
I20260928 15:03:32.586750 17766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:32.590907 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 17766
I20260928 15:03:32.591115 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:32.591697 9267 external_mini_cluster.cc:1428] Setting key 2880c45790e09835083696c661dda47b
Sep 28 15:03:32 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:32.609549 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.589389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:34473 (local address 127.9.12.254:33639)
0928 15:03:32.589589 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.589593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.590846 (+ 1253us) server_negotiation.cc:407] Connection header received
0928 15:03:32.593187 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.593191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.593255 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.593336 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.595297 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.595990 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.598532 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.599124 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.599251 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.603619 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.603623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.603626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:32.603657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.606292 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:32.607253 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:32.607257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:32.607259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:32.607399 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:32.607840 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:32.607843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:32.607845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:32.608097 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.608204 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.609111 (+ 907us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.609282 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:03:32.611074 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.589880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:34473)
0928 15:03:32.590640 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.590710 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.592884 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.593629 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.593638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.594327 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.595116 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.595146 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.596148 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.596156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.598262 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.598279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.599355 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.599364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.599708 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.600496 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:32.600534 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.603404 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.606417 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:32.606429 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:32.606449 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:32.607134 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:32.607486 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:32.607494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:32.607500 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:32.607732 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:32.608226 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.608236 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.608716 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.609019 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.609597 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
I20260928 15:03:32.612564 17894 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33639
I20260928 15:03:32.612861 17894 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.613574 17894 heartbeater.cc:511] Master 127.9.12.254:33639 requested a full tablet report, sending...
I20260928 15:03:32.615037 17571 ts_manager.cc:194] Registered new tserver with Master: 70fa94f109194edd8fc207024e115556 (127.9.12.194:45433)
I20260928 15:03:32.615986 17571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:34473
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:03:32.645366 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33639
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:32.962809 17899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.963091 17899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.963179 17899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.970203 17899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.970300 17899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.970402 17899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:32.979348 17899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:32.980584 17899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.981839 17899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.994817 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.994776 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.994861 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.995604 17899 server_base.cc:1034] running on GCE node
I20260928 15:03:32.996205 17899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.997140 17899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.998330 17899 hybrid_clock.cc:648] HybridClock initialized: now 1790607812998292 us; error 34 us; skew 500 ppm
I20260928 15:03:32.998641 17899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:32 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:33.001837 17899 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:03:33.003124 17899 webserver.cc:460] Webserver started at http://127.9.12.195:43343/ using document root <none> and password file <none>
I20260928 15:03:33.003729 17899 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:33.003854 17899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:33.004144 17899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:33.007011 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "b7e35bb4667b4e21b9fbc1c831fb26b1"
format_stamp: "Formatted at 2026-09-28 15:03:33 on dist-test-slave-dmhs"
server_key: "8ed71f9d008fefd5cd3530abc72b1414"
server_key_iv: "aad32c97515b8d40fdf3b5a95d9b586a"
server_key_version: "encryptionkey@0"
I20260928 15:03:33.007766 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "b7e35bb4667b4e21b9fbc1c831fb26b1"
format_stamp: "Formatted at 2026-09-28 15:03:33 on dist-test-slave-dmhs"
server_key: "8ed71f9d008fefd5cd3530abc72b1414"
server_key_iv: "aad32c97515b8d40fdf3b5a95d9b586a"
server_key_version: "encryptionkey@0"
I20260928 15:03:33.012655 17899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:03:33.016149 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:33.017428 17899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:33.017597 17899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "b7e35bb4667b4e21b9fbc1c831fb26b1"
format_stamp: "Formatted at 2026-09-28 15:03:33 on dist-test-slave-dmhs"
server_key: "8ed71f9d008fefd5cd3530abc72b1414"
server_key_iv: "aad32c97515b8d40fdf3b5a95d9b586a"
server_key_version: "encryptionkey@0"
I20260928 15:03:33.017809 17899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:33.037979 17899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:33.041652 17899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:33.041920 17899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:33.042809 17899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:33.044024 17899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:33.044132 17899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:33.044296 17899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:33.044368 17899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:33.075959 17899 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:36979
I20260928 15:03:33.076009 18026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:36979 every 8 connection(s)
I20260928 15:03:33.077337 17899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:33.087016 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 17899
I20260928 15:03:33.087215 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:33.087774 9267 external_mini_cluster.cc:1428] Setting key a4fd35b72aa5c5ffe71f1a81ed013e3e
Sep 28 15:03:33 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:33.097787 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:33.079864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:43027 (local address 127.9.12.254:33639)
0928 15:03:33.080076 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:33.080082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:33.081345 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 15:03:33.083870 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:33.083877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:33.083997 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:33.084113 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:33.086083 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.086855 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.088901 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.089386 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.089475 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:33.092927 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:33.092931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:33.092934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:33.092964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:33.095232 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.095823 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.095827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.095829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.095944 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.096387 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.096390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.096392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.096642 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:33.096749 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:33.097360 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 15:03:33.097514 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:03:33.098773 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:33.080351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:43027)
0928 15:03:33.081165 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:33.081212 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:33.083512 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:33.084375 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:33.084387 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:33.085146 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:33.085874 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:33.085903 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.087024 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.087028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:33.088748 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:33.088760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.089615 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.089621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:33.089803 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:33.090467 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:33.090504 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:33.092739 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:33.095340 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:33.095346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:33.095357 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:33.095727 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:33.096059 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:33.096062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:33.096064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:33.096224 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:33.096758 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:33.096763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:33.097048 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 15:03:33.097268 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 15:03:33.097633 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260928 15:03:33.100152 18027 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33639
I20260928 15:03:33.100561 18027 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:33.101253 18027 heartbeater.cc:511] Master 127.9.12.254:33639 requested a full tablet report, sending...
I20260928 15:03:33.102813 17571 ts_manager.cc:194] Registered new tserver with Master: b7e35bb4667b4e21b9fbc1c831fb26b1 (127.9.12.195:36979)
I20260928 15:03:33.103576 17571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:43027
I20260928 15:03:33.115314 17760 heartbeater.cc:503] Master 127.9.12.254:33639 was elected leader, sending a full tablet report...
I20260928 15:03:33.115481 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab.
Entry for principal oryx/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab.
WARNING: no policy specified for oryx/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab.
Entry for principal oryx/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab.
WARNING: no policy specified for oryx/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab.
Entry for principal oryx/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab.
I20260928 15:03:33.248149 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 17632
Sep 28 15:03:33 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:03:33.264195 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:33.249287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.9.12.193:43697)
0928 15:03:33.250135 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:33.250142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:33.250168 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:03:33.250232 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:33.250239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:33.250466 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:33.250616 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:33.251851 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.252954 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.254061 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.254565 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.254651 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:33.258645 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:33.258657 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:33.258670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:33.258702 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:33.261732 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.262361 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.262368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.262370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.262499 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.262812 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.262815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.262817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.263133 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:33.263352 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:33.263579 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 15:03:33.263769 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":519,"threads_started":1}
I20260928 15:03:33.265472 17755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:33.345708 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 17632
I20260928 15:03:33.362291 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 17766
Sep 28 15:03:33 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:03:33.377875 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:33.363178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.9.12.194:45433)
0928 15:03:33.363701 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:33.363708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:33.363733 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:03:33.363793 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:33.363799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:33.363996 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:33.364181 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:33.365480 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.366492 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.367792 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.368521 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.368658 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:33.372804 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:33.372815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:33.372829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:33.372869 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:33.375469 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.376064 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.376071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.376073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.376193 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.376575 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.376578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.376580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.376889 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:33.377119 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:33.377389 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:03:33.377580 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":275,"threads_started":1}
I20260928 15:03:33.379104 17889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:33.465624 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 17766
I20260928 15:03:33.481324 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 17899
Sep 28 15:03:33 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:03:33.497155 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:33.482381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.9.12.195:36979)
0928 15:03:33.483018 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:33.483025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:33.483052 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:03:33.483110 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:33.483116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:33.483324 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:33.483441 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:33.484731 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.485761 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.487070 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:33.487600 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:33.487694 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:33.491846 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:33.491856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:33.491870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:33.491905 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:33.494623 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.495298 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.495304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.495307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.495430 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:33.495736 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:33.495740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:33.495742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:33.496068 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:33.496374 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:33.496572 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 15:03:33.496763 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":347,"threads_started":1}
I20260928 15:03:33.498432 18022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:33.579651 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 17899
I20260928 15:03:33.595250 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 17539
I20260928 15:03:33.596514 17601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:33.696210 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 17539
I20260928 15:03:33.716261 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33639
--webserver_interface=127.9.12.254
--webserver_port=40773
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:33639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:34.028465 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:34.028738 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:34.028901 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:34.035754 18044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:34.035907 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:34.036018 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:34.036140 18044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:34.036211 18044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:34.045423 18044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:33639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=40773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:34.046718 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:34.048030 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:34.059355 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.059396 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.059285 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:34.060316 18044 server_base.cc:1034] running on GCE node
I20260928 15:03:34.061084 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:34.062461 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:34.063663 18044 hybrid_clock.cc:648] HybridClock initialized: now 1790607814063626 us; error 40 us; skew 500 ppm
I20260928 15:03:34.064018 18044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:34.067121 18044 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:03:34.068641 18044 webserver.cc:460] Webserver started at http://127.9.12.254:40773/ using document root <none> and password file <none>
I20260928 15:03:34.069314 18044 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:34.069464 18044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:34.074539 18044 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 15:03:34.077211 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.078395 18044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:03:34.078572 18044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "c1ae704b49cf48d3a38267234d355453"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "054aa51bf1b31acd0c20e785cb28492e"
server_key_iv: "dcab3dcb3a98903cb458286c13f9ddc1"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.079504 18044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:34.098150 18044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:34.102375 18044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:34.102626 18044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:34.120239 18044 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:33639
I20260928 15:03:34.120276 18110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:33639 every 8 connection(s)
I20260928 15:03:34.121632 18044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:34.128176 18111 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:03:34.128341 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18044
I20260928 15:03:34.129297 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:43697
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:33639
--webserver_port=37545
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:34.133131 18111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Bootstrap starting.
I20260928 15:03:34.152912 18111 log.cc:824] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:34.161613 18111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:03:34.162127 18111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Bootstrap complete.
I20260928 15:03:34.171025 18111 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:34.171845 18111 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ae704b49cf48d3a38267234d355453, State: Initialized, Role: FOLLOWER
I20260928 15:03:34.172534 18111 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:34.172770 18111 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:34.172876 18111 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:34.173072 18111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:03:34.176762 18111 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:34.177271 18111 leader_election.cc:302] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1ae704b49cf48d3a38267234d355453; no voters:
I20260928 15:03:34.177948 18111 leader_election.cc:288] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:03:34.178082 18115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:03:34.179436 18115 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [term 2 LEADER]: Becoming Leader. State: Replica: c1ae704b49cf48d3a38267234d355453, State: Running, Role: LEADER
I20260928 15:03:34.180002 18115 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } }
I20260928 15:03:34.180203 18111 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:34.182369 18117 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ae704b49cf48d3a38267234d355453. Latest consensus state: current_term: 2 leader_uuid: "c1ae704b49cf48d3a38267234d355453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } } }
I20260928 15:03:34.182734 18117 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:34.183193 18116 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1ae704b49cf48d3a38267234d355453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ae704b49cf48d3a38267234d355453" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33639 } } }
I20260928 15:03:34.183475 18116 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:34.185441 18120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:34.192327 18120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:34.194121 18120 catalog_manager.cc:1259] Loaded cluster ID: 37620e29ad564cd5b8ff99153fa436d4
I20260928 15:03:34.194298 18120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:34.199432 18120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:34.201653 18120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1ae704b49cf48d3a38267234d355453: Loaded TSK: 0
I20260928 15:03:34.203437 18120 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:03:34.491216 18113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:34.491549 18113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:34.491717 18113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:34.499051 18113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:34.499204 18113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:34.499375 18113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:34.509109 18113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.193.keytab
--principal=oryx/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:43697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=37545
--tserver_master_addrs=127.9.12.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:34.510456 18113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:34.511770 18113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:34.526096 18140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.526275 18138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.526660 18137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:34.527086 18113 server_base.cc:1034] running on GCE node
I20260928 15:03:34.527842 18113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:34.529065 18113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:34.530319 18113 hybrid_clock.cc:648] HybridClock initialized: now 1790607814530260 us; error 50 us; skew 500 ppm
I20260928 15:03:34.530694 18113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:34.534700 18113 init.cc:377] Logged in from keytab as oryx/127.9.12.193@KRBTEST.COM (short username oryx)
I20260928 15:03:34.536556 18113 webserver.cc:460] Webserver started at http://127.9.12.193:37545/ using document root <none> and password file <none>
I20260928 15:03:34.537457 18113 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:34.537608 18113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:34.544533 18113 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 15:03:34.547797 18147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.549500 18113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:34.549672 18113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "05ecae8485f743deabd847d1764adb19"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-dmhs"
server_key: "49bacc7e0b11705e5ddf52ed6c31aa6a"
server_key_iv: "01033f29c869c42403b9e2ac20780ea5"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.551029 18113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:34.578428 18113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:34.583662 18113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:34.583921 18113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:34.585084 18113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:34.586946 18113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:34.587061 18113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.587170 18113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:34.587289 18113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:34.620764 18113 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:43697
I20260928 15:03:34.620847 18259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:43697 every 8 connection(s)
I20260928 15:03:34.622179 18113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:34.624027 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18113
I20260928 15:03:34.624699 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:45433
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:33639
--webserver_port=42463
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:34.642743 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.625527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:59791)
0928 15:03:34.628894 (+ 3367us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:34.628947 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:34.630627 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:34.632031 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:34.632043 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:34.632667 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:34.633158 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.633174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.634723 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.634733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.635898 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.635910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.637010 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.637017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.637282 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.637961 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:34.637986 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:34.641546 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3084,"thread_start_us":2920,"threads_started":1}
W20260928 15:03:34.642863 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:34.625403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:59791 (local address 127.9.12.254:33639)
0928 15:03:34.626224 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:34.626242 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:34.629127 (+ 2885us) server_negotiation.cc:407] Connection header received
0928 15:03:34.631192 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:34.631205 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:34.631532 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:34.631880 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:34.633360 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.634552 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.636047 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.636764 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.636974 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.641912 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:59791: BlockingRecv error: recv got EOF from 127.9.12.193:59791 (error 108)
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":286,"threads_started":1}
W20260928 15:03:34.643402 18260 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:33639 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.12.254:33639: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:34.652957 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:34.644126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:48145 (local address 127.9.12.254:33639)
0928 15:03:34.644350 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:34.644354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:34.644369 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:03:34.644459 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:34.644463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:34.644512 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:34.644583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:34.645456 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.646139 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.647975 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.648462 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.648562 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.652777 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:48145: BlockingRecv error: recv got EOF from 127.9.12.193:48145 (error 108)
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":103}
I20260928 15:03:34.652974 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.643956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:48145)
0928 15:03:34.644194 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:34.644216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:34.644347 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:34.644650 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:34.644657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:34.645089 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:34.645301 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.645314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.646309 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.646314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.647789 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.647804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.648604 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.648610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.648814 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.649361 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:34.649383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:34.652621 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:34.661911 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.653681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:39015)
0928 15:03:34.653869 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:34.653887 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:34.653989 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:34.654456 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:34.654460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:34.654772 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:34.654928 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.654936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.656364 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.656368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.657452 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.657464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.658317 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.658321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.658461 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.659098 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:34.659115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:34.661702 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:03:34.662173 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:34.653958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:39015 (local address 127.9.12.254:33639)
0928 15:03:34.654135 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:34.654141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:34.654159 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:34.654207 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:34.654213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:34.654294 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:34.654414 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:34.655052 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.656216 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.657589 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.658159 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.658284 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.661939 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:39015: BlockingRecv error: recv got EOF from 127.9.12.193:39015 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 15:03:34.662328 18260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:03:34.946563 18264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:34.946862 18264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:34.947057 18264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:34.954015 18264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:34.954160 18264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:34.954326 18264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:34.963855 18264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.194.keytab
--principal=oryx/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:45433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=42463
--tserver_master_addrs=127.9.12.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:34.965166 18264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:34.966449 18264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:34.978883 18269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.978950 18270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.979074 18272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:34.979789 18264 server_base.cc:1034] running on GCE node
I20260928 15:03:34.980429 18264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:34.981415 18264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:34.982628 18264 hybrid_clock.cc:648] HybridClock initialized: now 1790607814982591 us; error 75 us; skew 500 ppm
I20260928 15:03:34.982920 18264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:34 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:34.991652 18264 init.cc:377] Logged in from keytab as oryx/127.9.12.194@KRBTEST.COM (short username oryx)
I20260928 15:03:34.993125 18264 webserver.cc:460] Webserver started at http://127.9.12.194:42463/ using document root <none> and password file <none>
I20260928 15:03:34.993794 18264 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:34.993954 18264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:34.999094 18264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 15:03:35.001705 18279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.002920 18264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:03:35.003093 18264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "70fa94f109194edd8fc207024e115556"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-dmhs"
server_key: "02aaee7dbacab21f221cbcec4bf78e51"
server_key_iv: "0d352f1a8e081dc9f9a4b306a9bb8cf6"
server_key_version: "encryptionkey@0"
I20260928 15:03:35.003995 18264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:35.018185 18264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:35.021816 18264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:35.022087 18264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:35.022984 18264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:35.024207 18264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:35.024364 18264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.024487 18264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:35.024578 18264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.056646 18264 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:45433
I20260928 15:03:35.056728 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:45433 every 8 connection(s)
I20260928 15:03:35.058115 18264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:35.065507 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18264
I20260928 15:03:35.066159 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:36979
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:33639
--webserver_port=43343
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:35.075148 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.060637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:39801 (local address 127.9.12.254:33639)
0928 15:03:35.060814 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.060818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.061859 (+ 1041us) server_negotiation.cc:407] Connection header received
0928 15:03:35.063581 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.063585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.063650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.063734 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.065184 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.065922 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.068506 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.068911 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.069018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.074964 (+ 5946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:39801: BlockingRecv error: recv got EOF from 127.9.12.194:39801 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:03:35.076067 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.061019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:39801)
0928 15:03:35.061677 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.061712 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.063344 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.063930 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.063938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.064496 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.065024 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.065040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.066081 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.066086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.068362 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.068380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.069140 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.069149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.069462 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.070189 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.070221 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.074812 (+ 4591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":260,"threads_started":1}
W20260928 15:03:35.076692 18392 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:33639 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.12.254:33639: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:35.085733 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.077266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:40637)
0928 15:03:35.077441 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.077464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.077559 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.077860 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.077865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.078189 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.078316 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.078324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.079252 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.079257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.080230 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.080241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.081403 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.081407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.081529 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.082131 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.082148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.085376 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:03:35.085772 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.077389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:40637 (local address 127.9.12.254:33639)
0928 15:03:35.077588 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.077596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.077617 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:03:35.077666 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.077671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.077741 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.077842 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.078450 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.079118 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.080495 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.081266 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.081411 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.085550 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:40637: BlockingRecv error: recv got EOF from 127.9.12.194:40637 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:35.093938 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.086506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:43739)
0928 15:03:35.086687 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.086707 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.086802 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.087116 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.087121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.087476 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.087613 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.087621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.088635 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.088640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.089537 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.089547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.090252 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.090258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.090465 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.091012 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.091028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.093668 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:03:35.094090 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.086588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:43739 (local address 127.9.12.254:33639)
0928 15:03:35.086791 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.086797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.086814 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:03:35.086912 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.086917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.086987 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.087087 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.087746 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.088508 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.089679 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.090106 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.090232 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.093879 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:43739: BlockingRecv error: recv got EOF from 127.9.12.194:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 15:03:35.094226 18392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:03:35.384680 18395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:35.384961 18395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:35.385054 18395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:35.392127 18395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:35.392225 18395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:35.392405 18395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:35.401854 18395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/krb5kdc/oryx_127.9.12.195.keytab
--principal=oryx/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:36979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=43343
--tserver_master_addrs=127.9.12.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:35.403039 18395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:35.404692 18395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:35.416872 18400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:35.417618 18401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:35.418025 18403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:35.418754 18395 server_base.cc:1034] running on GCE node
I20260928 15:03:35.419402 18395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:35.420243 18395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:35.421506 18395 hybrid_clock.cc:648] HybridClock initialized: now 1790607815421467 us; error 59 us; skew 500 ppm
I20260928 15:03:35.421798 18395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607815, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:35.424868 18395 init.cc:377] Logged in from keytab as oryx/127.9.12.195@KRBTEST.COM (short username oryx)
I20260928 15:03:35.426187 18395 webserver.cc:460] Webserver started at http://127.9.12.195:43343/ using document root <none> and password file <none>
I20260928 15:03:35.426896 18395 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:35.427043 18395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:35.432238 18395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:03:35.434703 18410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.435858 18395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:35.436020 18395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "b7e35bb4667b4e21b9fbc1c831fb26b1"
format_stamp: "Formatted at 2026-09-28 15:03:33 on dist-test-slave-dmhs"
server_key: "8ed71f9d008fefd5cd3530abc72b1414"
server_key_iv: "aad32c97515b8d40fdf3b5a95d9b586a"
server_key_version: "encryptionkey@0"
I20260928 15:03:35.437013 18395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:35.489204 18395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:35.493252 18395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:35.493520 18395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:35.494488 18395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:35.495721 18395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:35.495831 18395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.495956 18395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:35.496047 18395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.527578 18395 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:36979
I20260928 15:03:35.527664 18522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:36979 every 8 connection(s)
I20260928 15:03:35.529124 18395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:35.531237 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18395
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:35.545301 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.531813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:34947 (local address 127.9.12.254:33639)
0928 15:03:35.532036 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.532042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.533703 (+ 1661us) server_negotiation.cc:407] Connection header received
0928 15:03:35.535355 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.535360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.535463 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.535590 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.537062 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.537856 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.539487 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.539882 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.539997 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.545054 (+ 5057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:34947: BlockingRecv error: recv got EOF from 127.9.12.195:34947 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:03:35.545928 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.532484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:34947)
0928 15:03:35.533511 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.533564 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.535115 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.535809 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.535817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.536375 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.536907 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.536923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.538009 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.538013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.539353 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.539364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.540109 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.540119 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.540364 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.540906 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.540938 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.544855 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":351,"threads_started":1}
W20260928 15:03:35.546607 18523 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:33639 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.12.254:33639: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
I20260928 15:03:35.547374 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.532296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.9.12.254:33639)
0928 15:03:35.533892 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.533896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.533914 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:35.533961 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.533965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.534017 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.534123 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.535557 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.536361 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.537643 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.538068 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.538203 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.540772 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:35.540783 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:35.540802 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:35.540851 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:35.544175 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:35.544836 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:35.544843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:35.544848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:35.545011 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:35.545376 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:35.545382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:35.545387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:35.545815 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:35.546128 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.546582 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.546960 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1522,"thread_start_us":1419,"threads_started":1}
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:35.554865 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.547174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:39333)
0928 15:03:35.547358 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.547377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.547492 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.547828 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.547834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.548201 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.548400 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.548420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.549800 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.549805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.550883 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.550893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.551608 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.551612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.551724 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.552187 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.552202 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.554690 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:03:35.555037 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.547246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:39333 (local address 127.9.12.254:33639)
0928 15:03:35.547386 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.547390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.547568 (+ 178us) server_negotiation.cc:407] Connection header received
0928 15:03:35.547617 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.547623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.547695 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.547770 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.548567 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.549634 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.551048 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.551493 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.551588 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.554847 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:39333: BlockingRecv error: recv got EOF from 127.9.12.195:39333 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:35.562822 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.555553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:59033)
0928 15:03:35.555708 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.555725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.555813 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.556096 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.556100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.556493 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.556623 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.556630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.557572 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.557580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.558449 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.558458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.559084 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.559089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.559252 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.559710 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.559739 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.562563 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 15:03:35.563141 18523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:03:35.563171 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.555610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:59033 (local address 127.9.12.254:33639)
0928 15:03:35.555802 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.555807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.555823 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:03:35.555900 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.555915 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.555985 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.556077 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.556753 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.557451 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.558584 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.558956 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.559076 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.562759 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:59033: BlockingRecv error: recv got EOF from 127.9.12.195:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:35 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:35.671530 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.663095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:53963)
0928 15:03:35.663566 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.663584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.663721 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.664037 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.664044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.664576 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.664772 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.664781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.665747 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.665753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.666754 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.666765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.667527 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.667533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.667691 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.668354 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.668390 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.671277 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
W20260928 15:03:35.671715 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:35.663147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:53963 (local address 127.9.12.254:33639)
0928 15:03:35.663340 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.663347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.663657 (+ 310us) server_negotiation.cc:407] Connection header received
0928 15:03:35.663831 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.663838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.663899 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.663992 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.664902 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.665597 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.666913 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.667331 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.667470 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.671504 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:53963: BlockingRecv error: recv got EOF from 127.9.12.193:53963 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 15:03:36 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:36 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:36.102562 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:36.094787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:38163)
0928 15:03:36.095359 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:36.095385 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:36.095551 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:36.095853 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:36.095856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:36.096392 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:36.096593 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:36.096602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.097552 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.097555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:36.098462 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:36.098471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.099110 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.099114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:36.099225 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:36.099781 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:36.099797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:36.102336 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":333,"threads_started":1}
W20260928 15:03:36.102769 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:36.094863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:38163 (local address 127.9.12.254:33639)
0928 15:03:36.095073 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:36.095080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:36.095473 (+ 393us) server_negotiation.cc:407] Connection header received
0928 15:03:36.095668 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:36.095671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:36.095732 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:36.095809 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:36.096713 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.097416 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.098594 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.098974 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.099065 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:36.102522 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:38163: BlockingRecv error: recv got EOF from 127.9.12.194:38163 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 15:03:36 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:36 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:36.572041 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:36.563794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:37203)
0928 15:03:36.564407 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:36.564423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:36.564536 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:36.564802 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:36.564805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:36.565367 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:36.565552 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:36.565561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.566553 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.566557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:36.567568 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:36.567584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.568483 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.568489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:36.568660 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:36.569325 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:36.569342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:36.571829 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":352,"threads_started":1}
W20260928 15:03:36.572203 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:36.563901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:37203 (local address 127.9.12.254:33639)
0928 15:03:36.564115 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:36.564121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:36.564499 (+ 378us) server_negotiation.cc:407] Connection header received
0928 15:03:36.564617 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:36.564620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:36.564680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:36.564756 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:36.565686 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.566429 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.567735 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.568309 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.568420 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:36.572010 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:37203: BlockingRecv error: recv got EOF from 127.9.12.195:37203 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 15:03:36 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:36 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:36.682304 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:36.672622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:33863)
0928 15:03:36.673127 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:36.673146 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:36.673300 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:36.673652 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:36.673657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:36.674131 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:36.674321 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:36.674331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.675668 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.675672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:36.676606 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:36.676617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.677274 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.677278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:36.677388 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:36.678033 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:36.678051 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:36.681970 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":269,"threads_started":1}
W20260928 15:03:36.682431 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:36.672688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:33863 (local address 127.9.12.254:33639)
0928 15:03:36.672942 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:36.672951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:36.673229 (+ 278us) server_negotiation.cc:407] Connection header received
0928 15:03:36.673424 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:36.673429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:36.673499 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:36.673614 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:36.674505 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.675538 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.676727 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:36.677137 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:36.677237 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:36.682180 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:33863: BlockingRecv error: recv got EOF from 127.9.12.193:33863 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 15:03:37 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:37 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:37.111999 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:37.103593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:56125)
0928 15:03:37.104117 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:37.104132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:37.104356 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:37.104705 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:37.104709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:37.105155 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:37.105301 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:37.105310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.106345 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.106353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:37.107344 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:37.107356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.108141 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.108147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:37.108323 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:37.108922 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:37.108939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:37.111640 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":296,"threads_started":1}
W20260928 15:03:37.112058 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:37.103709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:56125 (local address 127.9.12.254:33639)
0928 15:03:37.103930 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:37.103938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:37.104219 (+ 281us) server_negotiation.cc:407] Connection header received
0928 15:03:37.104491 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:37.104501 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:37.104563 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:37.104678 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:37.105458 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.106210 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.107490 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.107974 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.108112 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:37.111842 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:56125: BlockingRecv error: recv got EOF from 127.9.12.194:56125 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 15:03:37 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:37 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:37.581130 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:37.573027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:44779)
0928 15:03:37.573514 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:37.573545 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:37.573656 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:37.573948 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:37.573951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:37.574328 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:37.574472 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:37.574479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.575458 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.575462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:37.576583 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:37.576595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.577438 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.577442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:37.577554 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:37.578135 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:37.578151 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:37.580843 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
W20260928 15:03:37.581359 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:37.573224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:44779 (local address 127.9.12.254:33639)
0928 15:03:37.573456 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:37.573482 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:37.573618 (+ 136us) server_negotiation.cc:407] Connection header received
0928 15:03:37.573751 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:37.573754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:37.573829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:37.573907 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:37.574600 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.575328 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.576704 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.577306 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.577400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:37.581090 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:44779: BlockingRecv error: recv got EOF from 127.9.12.195:44779 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 15:03:37 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:37 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:37.691622 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:37.683488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:35981)
0928 15:03:37.684012 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:37.684032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:37.684207 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:37.684596 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:37.684601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:37.684983 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:37.685162 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:37.685171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.686269 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.686276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:37.687218 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:37.687230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.687905 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.687911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:37.688099 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:37.688754 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:37.688771 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:37.691377 (+ 2606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
W20260928 15:03:37.691820 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:37.683586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:35981 (local address 127.9.12.254:33639)
0928 15:03:37.683824 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:37.683832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:37.684130 (+ 298us) server_negotiation.cc:407] Connection header received
0928 15:03:37.684381 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:37.684387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:37.684459 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:37.684572 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:37.685314 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.686133 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.687363 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:37.687761 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:37.687895 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:37.691570 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:35981: BlockingRecv error: recv got EOF from 127.9.12.193:35981 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 15:03:38 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:38 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:38.121539 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:38.113064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:58807)
0928 15:03:38.113582 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:38.113596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:38.113750 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:38.114120 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:38.114125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:38.114486 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:38.114691 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:38.114699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.115714 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.115732 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:38.116747 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:38.116759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.117450 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.117456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:38.117643 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:38.118273 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:38.118290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:38.121250 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":276,"threads_started":1}
W20260928 15:03:38.121706 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:38.113126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:58807 (local address 127.9.12.254:33639)
0928 15:03:38.113352 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:38.113359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:38.113682 (+ 323us) server_negotiation.cc:407] Connection header received
0928 15:03:38.113890 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:38.113894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:38.113954 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:38.114090 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:38.114836 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.115587 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.116893 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.117297 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.117459 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:38.121479 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:58807: BlockingRecv error: recv got EOF from 127.9.12.194:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 15:03:38 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:38 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:38.590503 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:38.582027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:33167)
0928 15:03:38.582540 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:38.582556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:38.582671 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:38.583012 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:38.583016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:38.583450 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:38.583669 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:38.583680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.584711 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.584715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:38.585724 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:38.585736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.586598 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.586602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:38.586725 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:38.587388 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:38.587406 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:38.590123 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":267,"threads_started":1}
W20260928 15:03:38.590548 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:38.582108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:33167 (local address 127.9.12.254:33639)
0928 15:03:38.582340 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:38.582347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:38.582633 (+ 286us) server_negotiation.cc:407] Connection header received
0928 15:03:38.582783 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:38.582787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:38.582858 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:38.582939 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:38.583818 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.584602 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.585864 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.586436 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.586546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:38.590322 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:33167: BlockingRecv error: recv got EOF from 127.9.12.195:33167 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 15:03:38 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:38 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:38.700698 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:38.692710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:44545)
0928 15:03:38.693177 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:38.693196 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:38.693310 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:38.693697 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:38.693701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:38.694088 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:38.694317 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:38.694328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.695469 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.695473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:38.696469 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:38.696487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.697106 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.697110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:38.697234 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:38.697804 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:38.697821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:38.700465 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":257,"threads_started":1}
W20260928 15:03:38.700899 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:38.692737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:44545 (local address 127.9.12.254:33639)
0928 15:03:38.692988 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:38.692997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:38.693297 (+ 300us) server_negotiation.cc:407] Connection header received
0928 15:03:38.693425 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:38.693430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:38.693512 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:38.693630 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:38.694466 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.695356 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.696594 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:38.696994 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:38.697103 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:38.700674 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:44545: BlockingRecv error: recv got EOF from 127.9.12.193:44545 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 15:03:39 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:39 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:39.130859 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:39.122512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:43897)
0928 15:03:39.123042 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:39.123057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:39.123207 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:39.123598 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:39.123604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:39.124024 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:39.124173 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:39.124182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.125305 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.125310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:39.126251 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:39.126288 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.127279 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.127283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:39.127422 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:39.127989 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:39.128006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:39.130649 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":264,"threads_started":1}
W20260928 15:03:39.131013 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:39.122589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:43897 (local address 127.9.12.254:33639)
0928 15:03:39.122829 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:39.122836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:39.123144 (+ 308us) server_negotiation.cc:407] Connection header received
0928 15:03:39.123345 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:39.123354 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:39.123443 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:39.123549 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:39.124327 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.125169 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.126430 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.127136 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.127260 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:39.130825 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:43897: BlockingRecv error: recv got EOF from 127.9.12.194:43897 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:03:39 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:39 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:39.599931 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:39.591553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:40529)
0928 15:03:39.592052 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:39.592068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:39.592176 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:39.592539 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:39.592542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:39.592887 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:39.593035 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:39.593043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.594287 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.594292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:39.595516 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:39.595528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.596183 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.596186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:39.596381 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:39.597028 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:39.597046 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:39.599714 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":286,"threads_started":1}
W20260928 15:03:39.600116 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:39.591657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:40529 (local address 127.9.12.254:33639)
0928 15:03:39.591821 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:39.591825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:39.592140 (+ 315us) server_negotiation.cc:407] Connection header received
0928 15:03:39.592306 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:39.592311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:39.592393 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:39.592476 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:39.593153 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.594137 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.595639 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.596034 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.596152 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:39.599893 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:40529: BlockingRecv error: recv got EOF from 127.9.12.195:40529 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 15:03:39 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:39 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:39.709872 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:39.701629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:46339)
0928 15:03:39.702209 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:39.702228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:39.702372 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:39.702734 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:39.702738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:39.703188 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:39.703362 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:39.703371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.704391 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.704397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:39.705290 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:39.705300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.706010 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.706016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:39.706196 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:39.706855 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:39.706872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:39.709637 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":278,"threads_started":1}
W20260928 15:03:39.710041 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:39.701671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:46339 (local address 127.9.12.254:33639)
0928 15:03:39.701930 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:39.701938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:39.702318 (+ 380us) server_negotiation.cc:407] Connection header received
0928 15:03:39.702505 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:39.702511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:39.702595 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:39.702701 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:39.703521 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.704235 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.705431 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:39.705858 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:39.705986 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:39.709828 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:46339: BlockingRecv error: recv got EOF from 127.9.12.193:46339 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 15:03:40 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:40 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:40.147217 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:40.131975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:60345)
0928 15:03:40.132532 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:40.132547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:40.132652 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:40.132936 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:40.132939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:40.133324 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:40.133478 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:40.133487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.134421 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.134424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:40.135313 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:40.135324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.136092 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.136096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:40.136251 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:40.143862 (+ 7611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:40.143916 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:40.146962 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":291,"threads_started":1}
W20260928 15:03:40.147408 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:40.132104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:60345 (local address 127.9.12.254:33639)
0928 15:03:40.132310 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:40.132318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:40.132622 (+ 304us) server_negotiation.cc:407] Connection header received
0928 15:03:40.132756 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:40.132760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:40.132816 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:40.132894 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:40.133600 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.134303 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.135440 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.135940 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.136064 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:40.147160 (+ 11096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:60345: BlockingRecv error: recv got EOF from 127.9.12.194:60345 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 15:03:40 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:40 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:40.608433 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:40.600876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:46941)
0928 15:03:40.601343 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:40.601358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:40.601463 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:40.601738 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:40.601741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:40.602132 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:40.602295 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:40.602303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.603164 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.603168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:40.603983 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:40.603992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.604631 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.604635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:40.604762 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:40.605290 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:40.605307 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:40.608124 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":267,"threads_started":1}
W20260928 15:03:40.608605 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:40.600965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:46941 (local address 127.9.12.254:33639)
0928 15:03:40.601222 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:40.601227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:40.601429 (+ 202us) server_negotiation.cc:407] Connection header received
0928 15:03:40.601552 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:40.601562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:40.601623 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:40.601697 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:40.602402 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.603052 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.604089 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.604522 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.604614 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:40.608353 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:46941: BlockingRecv error: recv got EOF from 127.9.12.195:46941 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 15:03:40 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:40 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:40.722627 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:40.710712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:38809)
0928 15:03:40.711321 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:40.711340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:40.711447 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:40.711739 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:40.711742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:40.712097 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:40.712251 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:40.712260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.713527 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.713532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:40.714383 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:40.714394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.715102 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.715109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:40.715282 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:40.719553 (+ 4271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:40.719569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:40.722268 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":398,"threads_started":1}
W20260928 15:03:40.722793 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:40.710949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:38809 (local address 127.9.12.254:33639)
0928 15:03:40.711182 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:40.711189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:40.711414 (+ 225us) server_negotiation.cc:407] Connection header received
0928 15:03:40.711548 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:40.711551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:40.711618 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:40.711696 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:40.712466 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.713381 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.714539 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:40.714934 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:40.715071 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:40.722538 (+ 7467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:38809: BlockingRecv error: recv got EOF from 127.9.12.193:38809 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 15:03:41 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:41 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:41.156509 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:41.148197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:51801)
0928 15:03:41.148709 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:41.148725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:41.148835 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:41.149118 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:41.149121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:41.149493 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:41.149652 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:41.149660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.150622 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.150625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:41.151892 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:41.151906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.152735 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.152739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:41.152860 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:41.153506 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:41.153524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:41.156181 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":306,"threads_started":1}
W20260928 15:03:41.156765 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:41.148249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:51801 (local address 127.9.12.254:33639)
0928 15:03:41.148523 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:41.148531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:41.148799 (+ 268us) server_negotiation.cc:407] Connection header received
0928 15:03:41.148929 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:41.148932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:41.148996 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:41.149074 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:41.149784 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.150501 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.152049 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.152595 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.152691 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:41.156452 (+ 3761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:51801: BlockingRecv error: recv got EOF from 127.9.12.194:51801 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 15:03:41 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:41 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:41.617324 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:41.609318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:32985)
0928 15:03:41.609786 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:41.609802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:41.609965 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:41.610264 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:41.610267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:41.610633 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:41.610816 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:41.610838 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.612029 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.612034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:41.613148 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:41.613160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.613824 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.613828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:41.613939 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:41.614487 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:41.614502 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:41.617102 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":266,"threads_started":1}
W20260928 15:03:41.617491 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:41.609430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:32985 (local address 127.9.12.254:33639)
0928 15:03:41.609693 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:41.609699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:41.609880 (+ 181us) server_negotiation.cc:407] Connection header received
0928 15:03:41.610065 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:41.610069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:41.610128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:41.610209 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:41.610946 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.611887 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.613273 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.613702 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.613793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:41.617275 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:32985: BlockingRecv error: recv got EOF from 127.9.12.195:32985 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 28 15:03:41 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:41 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:41.732223 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:41.723823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:36625)
0928 15:03:41.724448 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:41.724467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:41.724572 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:41.724853 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:41.724856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:41.725228 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:41.725384 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:41.725393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.726286 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.726289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:41.727226 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:41.727236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.727866 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.727870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:41.727980 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:41.728670 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:41.728693 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:41.731856 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":289,"threads_started":1}
W20260928 15:03:41.732299 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:41.723910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:36625 (local address 127.9.12.254:33639)
0928 15:03:41.724103 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:41.724107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:41.724533 (+ 426us) server_negotiation.cc:407] Connection header received
0928 15:03:41.724661 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:41.724665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:41.724723 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:41.724802 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:41.725496 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.726170 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.727339 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:41.727731 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:41.727817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:41.732066 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:36625: BlockingRecv error: recv got EOF from 127.9.12.193:36625 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 15:03:42 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:42 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:42.164774 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:42.157345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:57981)
0928 15:03:42.157897 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:42.157913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:42.158024 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:42.158337 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:42.158340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:42.158718 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:42.158882 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:42.158891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.159832 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.159835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:42.160777 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:42.160788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.161414 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.161418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:42.161528 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:42.162069 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:42.162086 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:42.164536 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
W20260928 15:03:42.164925 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:42.157431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:57981 (local address 127.9.12.254:33639)
0928 15:03:42.157598 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:42.157602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:42.157992 (+ 390us) server_negotiation.cc:407] Connection header received
0928 15:03:42.158126 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:42.158130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:42.158204 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:42.158282 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:42.159009 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.159720 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.160889 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.161293 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.161379 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:42.164712 (+ 3333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:57981: BlockingRecv error: recv got EOF from 127.9.12.194:57981 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 15:03:42 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:42 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:42.626334 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:42.618270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:42311)
0928 15:03:42.618773 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:42.618789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:42.618919 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:42.619208 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:42.619211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:42.619615 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:42.619786 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:42.619795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.620834 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.620837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:42.621765 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:42.621775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.622519 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.622524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:42.622696 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:42.623327 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:42.623343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:42.626033 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":221,"threads_started":1}
W20260928 15:03:42.626535 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:42.618334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:42311 (local address 127.9.12.254:33639)
0928 15:03:42.618552 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:42.618561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:42.618902 (+ 341us) server_negotiation.cc:407] Connection header received
0928 15:03:42.619026 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:42.619030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:42.619091 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:42.619169 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:42.619907 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.620722 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.621932 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.622365 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.622499 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:42.626277 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:42311: BlockingRecv error: recv got EOF from 127.9.12.195:42311 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 15:03:42 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:42 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:42.741575 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:42.733256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:40751)
0928 15:03:42.733788 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:42.733807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:42.733960 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:42.734288 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:42.734293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:42.734795 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:42.734989 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:42.735003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.736000 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.736005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:42.736989 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:42.737001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.737772 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.737778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:42.737951 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:42.738583 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:42.738599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:42.741337 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":287,"threads_started":1}
W20260928 15:03:42.741750 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:42.733450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:40751 (local address 127.9.12.254:33639)
0928 15:03:42.733671 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:42.733677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:42.733890 (+ 213us) server_negotiation.cc:407] Connection header received
0928 15:03:42.734081 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:42.734086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:42.734161 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:42.734256 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:42.735142 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.735857 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.737129 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:42.737584 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:42.737715 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:42.741533 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:40751: BlockingRecv error: recv got EOF from 127.9.12.193:40751 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 15:03:43 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:43 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:43.174273 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:43.165843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:33757)
0928 15:03:43.166329 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:43.166344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:43.166499 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:43.166850 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:43.166855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:43.167294 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:43.167441 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:43.167449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.169040 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.169047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:43.170014 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:43.170026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.170756 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.170760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:43.170875 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:43.171431 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:43.171448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:43.174053 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
W20260928 15:03:43.174430 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:43.165922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:33757 (local address 127.9.12.254:33639)
0928 15:03:43.166144 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:43.166152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:43.166430 (+ 278us) server_negotiation.cc:407] Connection header received
0928 15:03:43.166630 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:43.166635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:43.166701 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:43.166821 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:43.167595 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.168848 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.170139 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.170589 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.170701 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:43.174225 (+ 3524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:33757: BlockingRecv error: recv got EOF from 127.9.12.194:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 15:03:43 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:43 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:43.635803 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:43.627455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:40993)
0928 15:03:43.627919 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:43.627935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:43.628046 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:43.628458 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:43.628461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:43.628905 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:43.629076 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:43.629083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.630016 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.630020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:43.631121 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:43.631154 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.632248 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.632254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:43.632432 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:43.632967 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:43.632984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:43.635587 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":254,"threads_started":1}
W20260928 15:03:43.635985 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:43.627613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:40993 (local address 127.9.12.254:33639)
0928 15:03:43.627856 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:43.627862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:43.628015 (+ 153us) server_negotiation.cc:407] Connection header received
0928 15:03:43.628154 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:43.628158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:43.628228 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:43.628314 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:43.629222 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.629899 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.631295 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.632024 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.632143 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:43.635756 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:40993: BlockingRecv error: recv got EOF from 127.9.12.195:40993 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 15:03:43 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:43 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:43.750722 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:43.742474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:49115)
0928 15:03:43.743022 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:43.743042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:43.743163 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:43.743446 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:43.743449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:43.743801 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:43.743959 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:43.743967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.744889 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.744893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:43.745795 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:43.745804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.746424 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.746428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:43.746544 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:43.747194 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:43.747211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:43.750481 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
W20260928 15:03:43.751008 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:43.742579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:49115 (local address 127.9.12.254:33639)
0928 15:03:43.742777 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:43.742781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:43.743119 (+ 338us) server_negotiation.cc:407] Connection header received
0928 15:03:43.743260 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:43.743264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:43.743319 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:43.743403 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:43.744076 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.744777 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.745912 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:43.746304 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:43.746398 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:43.750669 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:49115: BlockingRecv error: recv got EOF from 127.9.12.193:49115 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 15:03:44 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:44 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:44.183260 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:44.175341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:56803)
0928 15:03:44.175820 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:44.175836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:44.175954 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:44.176232 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:44.176235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:44.176613 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:44.176772 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:44.176780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.177765 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.177768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:44.178714 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:44.178724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.179332 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.179337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:44.179450 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:44.179974 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:44.179990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:44.182903 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
W20260928 15:03:44.183327 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:44.175439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:56803 (local address 127.9.12.254:33639)
0928 15:03:44.175693 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:44.175699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:44.175915 (+ 216us) server_negotiation.cc:407] Connection header received
0928 15:03:44.176051 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:44.176054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:44.176114 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:44.176190 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:44.176891 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.177660 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.178840 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.179218 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.179298 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:44.183113 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:56803: BlockingRecv error: recv got EOF from 127.9.12.194:56803 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 28 15:03:44 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:44 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:44.644748 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:44.636783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:51409)
0928 15:03:44.637293 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:44.637309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:44.637472 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:44.637805 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:44.637810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:44.638207 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:44.638373 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:44.638380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.639388 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.639395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:44.640434 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:44.640445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.641119 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.641125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:44.641296 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:44.641839 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:44.641855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:44.644512 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":275,"threads_started":1}
W20260928 15:03:44.644886 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:44.636828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:51409 (local address 127.9.12.254:33639)
0928 15:03:44.637047 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:44.637056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:44.637393 (+ 337us) server_negotiation.cc:407] Connection header received
0928 15:03:44.637618 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:44.637622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:44.637681 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:44.637779 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:44.638521 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.639263 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.640577 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.640967 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.641106 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:44.644677 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:51409: BlockingRecv error: recv got EOF from 127.9.12.195:51409 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 15:03:44 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:44 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:44.760352 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:44.751802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:52445)
0928 15:03:44.752382 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:44.752410 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:44.752554 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:44.752948 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:44.752954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:44.753489 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:44.753681 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:44.753693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.754904 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.754908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:44.755835 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:44.755846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.756641 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.756645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:44.756770 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:44.757341 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:44.757358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:44.760064 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":337,"threads_started":1}
W20260928 15:03:44.760568 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:44.751901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:52445 (local address 127.9.12.254:33639)
0928 15:03:44.752112 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:44.752116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:44.752490 (+ 374us) server_negotiation.cc:407] Connection header received
0928 15:03:44.752704 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:44.752709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:44.752790 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:44.752902 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:44.753869 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.754766 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.755959 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:44.756501 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:44.756594 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:44.760320 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:52445: BlockingRecv error: recv got EOF from 127.9.12.193:52445 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 15:03:45 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:45 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:45.198594 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:45.184342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:54029)
0928 15:03:45.184805 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:45.184820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:45.184959 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:45.185260 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:45.185264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:45.185692 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:45.185831 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:45.185838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.187003 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.187011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:45.188006 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:45.188018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.188801 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.188806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:45.188996 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:45.195422 (+ 6426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:45.195441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:45.198371 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
W20260928 15:03:45.198787 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:45.184382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:54029 (local address 127.9.12.254:33639)
0928 15:03:45.184575 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:45.184581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:45.184895 (+ 314us) server_negotiation.cc:407] Connection header received
0928 15:03:45.185070 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:45.185074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:45.185144 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:45.185235 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:45.185983 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.186852 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.188150 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.188650 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.188786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:45.198564 (+ 9778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:54029: BlockingRecv error: recv got EOF from 127.9.12.194:54029 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 15:03:45 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:45 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:45.654740 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:45.645744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:37133)
0928 15:03:45.646317 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:45.646334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:45.646475 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:45.646834 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:45.646838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:45.647277 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:45.647421 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:45.647429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.648710 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.648716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:45.649694 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:45.649707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.650444 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.650450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:45.650653 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:45.651370 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:45.651389 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:45.654495 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":294,"threads_started":1}
W20260928 15:03:45.655172 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:45.645917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:37133 (local address 127.9.12.254:33639)
0928 15:03:45.646150 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:45.646156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:45.646421 (+ 265us) server_negotiation.cc:407] Connection header received
0928 15:03:45.646606 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:45.646611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:45.646695 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:45.646800 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:45.647584 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.648550 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.649851 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.650273 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.650419 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:45.654736 (+ 4317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:37133: BlockingRecv error: recv got EOF from 127.9.12.195:37133 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 15:03:45 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:45 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:45.769733 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:45.761276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:43033)
0928 15:03:45.761827 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:45.761854 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:45.761967 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:45.762275 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:45.762278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:45.762682 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:45.762862 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:45.762870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.763816 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.763820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:45.764781 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:45.764797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.765787 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.765794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:45.765963 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:45.766649 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:45.766666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:45.769480 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":320,"threads_started":1}
W20260928 15:03:45.769971 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:45.761412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:43033 (local address 127.9.12.254:33639)
0928 15:03:45.761612 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:45.761616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:45.761927 (+ 311us) server_negotiation.cc:407] Connection header received
0928 15:03:45.762064 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:45.762067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:45.762152 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:45.762235 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:45.762984 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.763704 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.764954 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:45.765629 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:45.765764 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:45.769703 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:43033: BlockingRecv error: recv got EOF from 127.9.12.193:43033 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 15:03:46 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:46 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:46.207598 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:46.199600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:41631)
0928 15:03:46.200099 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:46.200114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:46.200229 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:46.200611 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:46.200622 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:46.201000 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:46.201169 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:46.201177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.202133 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.202136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:46.203008 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:46.203018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.203632 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.203636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:46.203743 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:46.204309 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:46.204325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:46.207228 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
W20260928 15:03:46.207690 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:46.199627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:41631 (local address 127.9.12.254:33639)
0928 15:03:46.199783 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:46.199787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:46.200187 (+ 400us) server_negotiation.cc:407] Connection header received
0928 15:03:46.200390 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:46.200393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:46.200451 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:46.200535 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:46.201306 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.202019 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.203115 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.203517 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.203610 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:46.207444 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:41631: BlockingRecv error: recv got EOF from 127.9.12.194:41631 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 15:03:46 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:46 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:46.663761 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:46.655781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:51061)
0928 15:03:46.656378 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:46.656393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:46.656508 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:46.656883 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:46.656887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:46.657474 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:46.657657 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:46.657666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.658621 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.658625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:46.659502 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:46.659512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.660136 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.660140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:46.660261 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:46.660963 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:46.660981 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:46.663530 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":340,"threads_started":1}
W20260928 15:03:46.663905 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:46.655806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:51061 (local address 127.9.12.254:33639)
0928 15:03:46.656006 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:46.656011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:46.656482 (+ 471us) server_negotiation.cc:407] Connection header received
0928 15:03:46.656601 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:46.656608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:46.656689 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:46.656820 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:46.657787 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.658515 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.659619 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.660009 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.660096 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:46.663710 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:51061: BlockingRecv error: recv got EOF from 127.9.12.195:51061 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 15:03:46 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:46 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:46.779263 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:46.770806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:32949)
0928 15:03:46.771323 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:46.771342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:46.771496 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:46.771876 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:46.771882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:46.772337 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:46.772490 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:46.772499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.773740 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.773748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:46.774759 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:46.774770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.775496 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.775502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:46.775672 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:46.776307 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:46.776324 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:46.779021 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
W20260928 15:03:46.779462 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:46.770877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:32949 (local address 127.9.12.254:33639)
0928 15:03:46.771120 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:46.771128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:46.771431 (+ 303us) server_negotiation.cc:407] Connection header received
0928 15:03:46.771652 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:46.771656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:46.771715 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:46.771826 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:46.772691 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.773588 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.774890 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:46.775352 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:46.775476 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:46.779212 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:32949: BlockingRecv error: recv got EOF from 127.9.12.193:32949 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 15:03:47 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:47 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:47.216637 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:47.208595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:32791)
0928 15:03:47.209062 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:47.209075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:47.209244 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:47.209598 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:47.209602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:47.209996 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:47.210145 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:47.210153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.211271 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.211278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:47.212154 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:47.212165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.212896 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.212902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:47.213098 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:47.213711 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:47.213728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:47.216273 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
W20260928 15:03:47.216729 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:47.208657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:32791 (local address 127.9.12.254:33639)
0928 15:03:47.208851 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:47.208859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:47.209166 (+ 307us) server_negotiation.cc:407] Connection header received
0928 15:03:47.209367 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:47.209372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:47.209442 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:47.209542 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:47.210291 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.211133 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.212315 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.212736 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.212884 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:47.216503 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:32791: BlockingRecv error: recv got EOF from 127.9.12.194:32791 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 15:03:47 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:47 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:47.672884 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:47.664767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:54207)
0928 15:03:47.665253 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:47.665269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:47.665387 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:47.665681 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:47.665684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:47.666031 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:47.666181 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:47.666188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.667145 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.667149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:47.668063 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:47.668074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.668803 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.668807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:47.668936 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:47.669486 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:47.669503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:47.672586 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":265,"threads_started":1}
W20260928 15:03:47.673076 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:47.664797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:54207 (local address 127.9.12.254:33639)
0928 15:03:47.665014 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:47.665022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:47.665346 (+ 324us) server_negotiation.cc:407] Connection header received
0928 15:03:47.665485 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:47.665488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:47.665549 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:47.665633 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:47.666296 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.667021 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.668181 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.668690 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.668779 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:47.672793 (+ 4014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:54207: BlockingRecv error: recv got EOF from 127.9.12.195:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:03:47 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:47 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:47.788879 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:47.780248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:37359)
0928 15:03:47.780808 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:47.780826 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:47.780974 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:47.781377 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:47.781380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:47.781748 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:47.781902 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:47.781911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.782983 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.782988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:47.783947 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:47.783959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.784767 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.784772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:47.784884 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:47.785501 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:47.785518 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:47.788519 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":270,"threads_started":1}
W20260928 15:03:47.789004 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:47.780324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:37359 (local address 127.9.12.254:33639)
0928 15:03:47.780557 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:47.780565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:47.780906 (+ 341us) server_negotiation.cc:407] Connection header received
0928 15:03:47.781117 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:47.781121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:47.781180 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:47.781311 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:47.782078 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.782831 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.784064 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:47.784621 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:47.784709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:47.788759 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:37359: BlockingRecv error: recv got EOF from 127.9.12.193:37359 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 15:03:48 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:48 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:48.225692 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:48.217682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:47387)
0928 15:03:48.218250 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:48.218266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:48.218373 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:48.218686 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:48.218690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:48.219090 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:48.219251 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:48.219259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.220345 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.220349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:48.221225 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:48.221234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.221929 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.221934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:48.222055 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:48.222695 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:48.222713 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:48.225467 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":287,"threads_started":1}
W20260928 15:03:48.225849 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:48.217768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:47387 (local address 127.9.12.254:33639)
0928 15:03:48.217930 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:48.217934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:48.218337 (+ 403us) server_negotiation.cc:407] Connection header received
0928 15:03:48.218466 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:48.218470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:48.218532 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:48.218610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:48.219375 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.220199 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.221335 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.221752 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.221841 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:48.225652 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:47387: BlockingRecv error: recv got EOF from 127.9.12.194:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 15:03:48 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:48 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:48.683039 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:48.673867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:34479 (local address 127.9.12.254:33639)
0928 15:03:48.674118 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:48.674124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:48.674343 (+ 219us) server_negotiation.cc:407] Connection header received
0928 15:03:48.674474 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:48.674477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:48.674546 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:48.674615 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:48.675287 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.675943 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.677104 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.677529 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.677622 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:48.682765 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:34479: BlockingRecv error: recv got EOF from 127.9.12.195:34479 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 15:03:48.683550 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:48.673778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:34479)
0928 15:03:48.674248 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:48.674275 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:48.674379 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:48.674628 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:48.674630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:48.675015 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:48.675173 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:48.675181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.676069 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.676072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:48.676965 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:48.676976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.677675 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.677679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:48.677786 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:48.678447 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:48.678473 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:48.683240 (+ 4767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":244,"threads_started":1}
Sep 28 15:03:48 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:48 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:48.798835 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:48.789934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:45071)
0928 15:03:48.790523 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:48.790541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:48.790693 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:48.791015 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:48.791019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:48.791401 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:48.791592 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:48.791600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.792785 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.792791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:48.794295 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:48.794308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.795005 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.795011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:48.795209 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:48.795740 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:48.795755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:48.798494 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":259,"threads_started":1}
W20260928 15:03:48.798950 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:48.790042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:45071 (local address 127.9.12.254:33639)
0928 15:03:48.790372 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:48.790379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:48.790618 (+ 239us) server_negotiation.cc:407] Connection header received
0928 15:03:48.790824 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:48.790828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:48.790887 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:48.790983 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:48.791728 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.792651 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.794436 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:48.794855 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:48.794981 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:48.798704 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:45071: BlockingRecv error: recv got EOF from 127.9.12.193:45071 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 28 15:03:49 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:49 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:49.234935 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:49.226709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:56163)
0928 15:03:49.227228 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:49.227244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:49.227356 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:49.227694 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:49.227699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:49.228156 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:49.228350 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:49.228360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.229690 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.229693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:49.230592 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:49.230602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.231237 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.231241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:49.231361 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:49.231910 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:49.231926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:49.234694 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":295,"threads_started":1}
W20260928 15:03:49.235065 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:49.226824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:56163 (local address 127.9.12.254:33639)
0928 15:03:49.227011 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:49.227016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:49.227314 (+ 298us) server_negotiation.cc:407] Connection header received
0928 15:03:49.227468 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:49.227477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:49.227536 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:49.227616 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:49.228495 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.229545 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.230709 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.231122 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.231209 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:49.234869 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:56163: BlockingRecv error: recv got EOF from 127.9.12.194:56163 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:03:49 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:49 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.195@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:49.692991 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:49.684463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.195:44281)
0928 15:03:49.685025 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:49.685042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:49.685163 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:49.685493 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:49.685496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:49.685808 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:49.686048 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:49.686059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.687162 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.687170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:49.688466 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:49.688479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.689234 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.689239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:49.689358 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:49.689939 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:49.689955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:49.692771 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
W20260928 15:03:49.693265 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:49.684524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:44281 (local address 127.9.12.254:33639)
0928 15:03:49.684676 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:49.684680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:49.685121 (+ 441us) server_negotiation.cc:407] Connection header received
0928 15:03:49.685292 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:49.685298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:49.685368 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:49.685449 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:49.686170 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.687004 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.688645 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.689109 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.689205 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:49.692978 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.195:44281: BlockingRecv error: recv got EOF from 127.9.12.195:44281 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 15:03:49 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:49 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.193@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:49.808374 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:49.800085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.193:49379)
0928 15:03:49.800646 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:49.800666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:49.800769 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:49.801084 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:49.801087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:49.801480 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:49.801689 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:49.801699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.802638 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.802643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:49.803600 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:49.803612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.804203 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.804207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:49.804371 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:49.804923 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:49.804939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:49.808062 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":304,"threads_started":1}
W20260928 15:03:49.808519 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:49.800337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:49379 (local address 127.9.12.254:33639)
0928 15:03:49.800539 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:49.800543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:49.800734 (+ 191us) server_negotiation.cc:407] Connection header received
0928 15:03:49.800859 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:49.800862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:49.800958 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:49.801041 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:49.801819 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.802503 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.803729 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:49.804098 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:49.804184 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:49.808239 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.193:49379: BlockingRecv error: recv got EOF from 127.9.12.193:49379 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 15:03:50 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:50 dist-test-slave-dmhs krb5kdc[17526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.12.194@KRBTEST.COM for oryx/127.9.12.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:50.248543 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:50.236009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33639 (local address 127.9.12.194:59769)
0928 15:03:50.236585 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:50.236601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:50.236704 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:50.237005 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:50.237008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:50.237375 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:50.237583 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:50.237591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.238533 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.238536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:50.239702 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:50.239714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.240704 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.240708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:50.240825 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:50.245496 (+ 4671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:50.245515 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:50.248227 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.12.254:33639: Server oryx/127.9.12.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":341,"threads_started":1}
W20260928 15:03:50.248724 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:50.236246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:59769 (local address 127.9.12.254:33639)
0928 15:03:50.236440 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:50.236444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:50.236673 (+ 229us) server_negotiation.cc:407] Connection header received
0928 15:03:50.236800 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:50.236803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:50.236862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:50.236944 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:50.237710 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.238407 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.239842 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.240572 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.240672 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:50.248488 (+ 7816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.12.194:59769: BlockingRecv error: recv got EOF from 127.9.12.194:59769 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:03:50.534200 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18113
I20260928 15:03:50.545459 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:50.535244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.9.12.193:43697)
0928 15:03:50.535765 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:50.535771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:50.535795 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:03:50.535855 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:50.535860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:50.536086 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:50.536231 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:50.537598 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.538544 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.539651 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.540162 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.540251 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:50.542606 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:50.542613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:50.542624 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:50.542650 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:50.545015 (+ 2365us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.12.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:03:50.545246 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38854: Request ticket server kudu/127.9.12.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":263,"threads_started":1}
W20260928 15:03:50.545579 18573 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38854: Request ticket server kudu/127.9.12.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:03:50.545578 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18113: Not authorized: Client connection negotiation failed: client connection to 127.9.12.193:43697: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.12.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:03:50.545754 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18113
I20260928 15:03:50.562479 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18264
I20260928 15:03:50.574137 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:50.563506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.9.12.194:45433)
0928 15:03:50.564111 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:50.564118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:50.564146 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:03:50.564204 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:50.564210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:50.564498 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:50.564650 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:50.565875 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.566899 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.568098 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.568751 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.568858 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:50.571295 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:50.571302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:50.571315 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:50.571347 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:50.573611 (+ 2264us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.12.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:03:50.573849 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55472: Request ticket server kudu/127.9.12.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":296,"threads_started":1}
W20260928 15:03:50.574239 18574 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55472: Request ticket server kudu/127.9.12.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:03:50.574237 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18264: Not authorized: Client connection negotiation failed: client connection to 127.9.12.194:45433: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.12.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:03:50.574390 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18264
I20260928 15:03:50.590438 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18395
I20260928 15:03:50.602396 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:50.591456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.9.12.195:36979)
0928 15:03:50.592025 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:50.592032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:50.592056 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:03:50.592112 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:50.592118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:50.592407 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:50.592561 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:50.593745 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.594764 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.595913 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:50.596606 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:50.596779 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:50.599369 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:50.599377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:50.599389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:50.599448 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:50.601777 (+ 2329us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.12.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:03:50.602084 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33718: Request ticket server kudu/127.9.12.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":311,"threads_started":1}
W20260928 15:03:50.602475 18575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33718: Request ticket server kudu/127.9.12.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:03:50.602555 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18395: Not authorized: Client connection negotiation failed: client connection to 127.9.12.195:36979: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.12.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:03:50.602715 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18395
I20260928 15:03:50.618700 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18044
I20260928 15:03:50.619870 18106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:50.736430 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18044
2026-09-28T15:03:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20981 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T15:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:50Z Disabled control of system clock
I20260928 15:03:50.812759 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:41459
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:41459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:03:51.141371 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:51.141639 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:51.141793 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:51.148782 18580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:51.148886 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:51.148988 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:51.149092 18580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:51.149154 18580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:51.158025 18580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:41459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:51.159309 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:51.161162 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:51.172756 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:51.172780 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:51.172894 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:51.173579 18580 server_base.cc:1034] running on GCE node
I20260928 15:03:51.174353 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:51.175659 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:51.176855 18580 hybrid_clock.cc:648] HybridClock initialized: now 1790607831176824 us; error 28 us; skew 500 ppm
I20260928 15:03:51.177210 18580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:51.179396 18580 webserver.cc:460] Webserver started at http://127.9.12.254:38259/ using document root <none> and password file <none>
I20260928 15:03:51.180086 18580 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:51.180213 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:51.180603 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:51.183465 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29"
format_stamp: "Formatted at 2026-09-28 15:03:51 on dist-test-slave-dmhs"
server_key: "aa131b8676439ebed15efc071177e6d9"
server_key_iv: "d0e5ab90e897199a2ff9626abe79a4a4"
server_key_version: "encryptionkey@0"
I20260928 15:03:51.184245 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29"
format_stamp: "Formatted at 2026-09-28 15:03:51 on dist-test-slave-dmhs"
server_key: "aa131b8676439ebed15efc071177e6d9"
server_key_iv: "d0e5ab90e897199a2ff9626abe79a4a4"
server_key_version: "encryptionkey@0"
I20260928 15:03:51.189047 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:03:51.192234 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:51.193542 18580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:03:51.193727 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29"
format_stamp: "Formatted at 2026-09-28 15:03:51 on dist-test-slave-dmhs"
server_key: "aa131b8676439ebed15efc071177e6d9"
server_key_iv: "d0e5ab90e897199a2ff9626abe79a4a4"
server_key_version: "encryptionkey@0"
I20260928 15:03:51.193940 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:51.206619 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:51.207417 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:51.207707 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:51.225720 18580 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:41459
I20260928 15:03:51.225740 18645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:41459 every 8 connection(s)
I20260928 15:03:51.227077 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:51.230763 18646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:51.232803 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18580
I20260928 15:03:51.233000 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:51.233774 9267 external_mini_cluster.cc:1428] Setting key 803931ac5c69b494fb74d62d3b5dccf3
I20260928 15:03:51.242311 18646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29: Bootstrap starting.
I20260928 15:03:51.245182 18646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:51.246251 18646 log.cc:824] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:51.246855 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:51.235924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.9.12.254:41459)
0928 15:03:51.236921 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:51.236930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:51.236969 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:03:51.238540 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:51.238568 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:51.238977 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:51.239358 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:51.239943 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:51.241293 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:51.242664 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:51.243295 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:51.243497 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:51.244639 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:51.244646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:51.244659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:51.245127 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:51.245213 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:51.245257 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:51.245366 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:51.245571 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 15:03:51.245837 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":363,"threads_started":1}
I20260928 15:03:51.249241 18646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29: No bootstrap required, opened a new log
I20260928 15:03:51.256861 18646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41459 } }
I20260928 15:03:51.257185 18646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:51.257257 18646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0ac4b1805b473ea1846b9e2bc4ea29, State: Initialized, Role: FOLLOWER
I20260928 15:03:51.257791 18646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41459 } }
I20260928 15:03:51.257987 18646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:51.258092 18646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:51.258208 18646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:51.260316 18646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41459 } }
I20260928 15:03:51.260944 18646 leader_election.cc:302] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d0ac4b1805b473ea1846b9e2bc4ea29; no voters:
I20260928 15:03:51.261525 18646 leader_election.cc:288] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:51.261682 18651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:51.262491 18651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0ac4b1805b473ea1846b9e2bc4ea29, State: Running, Role: LEADER
I20260928 15:03:51.262969 18651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41459 } }
I20260928 15:03:51.263522 18646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:51.265760 18652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41459 } } }
I20260928 15:03:51.266251 18652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:51.265865 18653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d0ac4b1805b473ea1846b9e2bc4ea29. Latest consensus state: current_term: 1 leader_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0ac4b1805b473ea1846b9e2bc4ea29" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41459 } } }
I20260928 15:03:51.268046 18653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:51.268659 18660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:51.272724 18660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:51.281009 18660 catalog_manager.cc:1347] Generated new cluster ID: c8f9545cdffc4e60a1bbb2ee34cb0ba3
I20260928 15:03:51.281111 18660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:51.297802 18660 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:51.299155 18660 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:51.306510 18660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d0ac4b1805b473ea1846b9e2bc4ea29: Generated new TSK 0
I20260928 15:03:51.307304 18660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:51.316390 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41459
--builtin_ntp_servers=127.9.12.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 15:03:51.638444 18670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:51.638764 18670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:51.638916 18670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:51.646042 18670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:51.646126 18670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:51.646224 18670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:51.655318 18670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:51.656983 18670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:51.658648 18670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:51.671419 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:51.671454 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:51.671677 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:51.672531 18670 server_base.cc:1034] running on GCE node
I20260928 15:03:51.673089 18670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:51.673884 18670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:51.675073 18670 hybrid_clock.cc:648] HybridClock initialized: now 1790607831675037 us; error 25 us; skew 500 ppm
I20260928 15:03:51.675374 18670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:51.677450 18670 webserver.cc:460] Webserver started at http://127.9.12.193:33019/ using document root <none> and password file <none>
I20260928 15:03:51.678109 18670 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:51.678222 18670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:51.678545 18670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:51.681311 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "b602362653954c6d9e086a29db9d423f"
format_stamp: "Formatted at 2026-09-28 15:03:51 on dist-test-slave-dmhs"
server_key: "6af12d8f7f486bb0d251156d6e53cdbd"
server_key_iv: "706cf55aee0c0ba17e0dca73ae96410a"
server_key_version: "encryptionkey@0"
I20260928 15:03:51.682068 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "b602362653954c6d9e086a29db9d423f"
format_stamp: "Formatted at 2026-09-28 15:03:51 on dist-test-slave-dmhs"
server_key: "6af12d8f7f486bb0d251156d6e53cdbd"
server_key_iv: "706cf55aee0c0ba17e0dca73ae96410a"
server_key_version: "encryptionkey@0"
I20260928 15:03:51.686830 18670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:03:51.690165 18684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:51.691387 18670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:51.691577 18670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "b602362653954c6d9e086a29db9d423f"
format_stamp: "Formatted at 2026-09-28 15:03:51 on dist-test-slave-dmhs"
server_key: "6af12d8f7f486bb0d251156d6e53cdbd"
server_key_iv: "706cf55aee0c0ba17e0dca73ae96410a"
server_key_version: "encryptionkey@0"
I20260928 15:03:51.691795 18670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:51.712579 18670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:51.713374 18670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:51.713672 18670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:51.714587 18670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:51.715932 18670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:51.716095 18670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:51.716215 18670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:51.716326 18670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:51.748397 18670 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:38723
I20260928 15:03:51.748440 18796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:38723 every 8 connection(s)
I20260928 15:03:51.749691 18670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:51.751005 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18670
I20260928 15:03:51.751209 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:51.752022 9267 external_mini_cluster.cc:1428] Setting key 40db07a55562419af87b3f474479e797
I20260928 15:03:51.754951 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41459
--builtin_ntp_servers=127.9.12.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 15:03:51.762451 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:51.752526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:36939 (local address 127.9.12.254:41459)
0928 15:03:51.753019 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:51.753025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:51.754327 (+ 1302us) server_negotiation.cc:407] Connection header received
0928 15:03:51.756124 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:51.756129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:51.756218 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:51.756365 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:51.757383 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:51.758115 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:51.759647 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:51.760055 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:51.760190 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:51.761733 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:51.761737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:51.761741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:51.761757 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:51.761806 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:51.761817 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:51.761897 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:51.762140 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 15:03:51.762216 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260928 15:03:51.763876 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:51.753128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:41459 (local address 127.9.12.193:36939)
0928 15:03:51.754175 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:51.754218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:51.755874 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:51.756586 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:51.756593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:51.756677 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:51.757222 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:51.757239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:51.758276 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:51.758281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:51.759496 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:51.759511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:51.760302 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:51.760313 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:51.760575 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:51.761351 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:51.761356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:51.761415 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:51.761517 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:51.761921 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:51.761934 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:51.761945 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:03:51.762127 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 15:03:51.762531 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":295,"threads_started":1}
I20260928 15:03:51.765487 18797 heartbeater.cc:347] Connected to a master server at 127.9.12.254:41459
I20260928 15:03:51.765797 18797 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:51.766674 18797 heartbeater.cc:511] Master 127.9.12.254:41459 requested a full tablet report, sending...
I20260928 15:03:51.769042 18611 ts_manager.cc:194] Registered new tserver with Master: b602362653954c6d9e086a29db9d423f (127.9.12.193:38723)
I20260928 15:03:51.771180 18611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.193:36939
W20260928 15:03:52.071933 18801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:52.072175 18801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:52.072254 18801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:52.079792 18801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:52.079876 18801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:52.079991 18801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:52.088464 18801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:52.089541 18801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:52.091114 18801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:52.104162 18806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:52.104050 18809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:52.104050 18807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:52.104810 18801 server_base.cc:1034] running on GCE node
I20260928 15:03:52.105417 18801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:52.106339 18801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:52.107599 18801 hybrid_clock.cc:648] HybridClock initialized: now 1790607832107561 us; error 61 us; skew 500 ppm
I20260928 15:03:52.107932 18801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:52.110220 18801 webserver.cc:460] Webserver started at http://127.9.12.194:44473/ using document root <none> and password file <none>
I20260928 15:03:52.110872 18801 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:52.111006 18801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:52.111311 18801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:52.114034 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "aee9bae3b8654923a54d64ccc0a02b70"
format_stamp: "Formatted at 2026-09-28 15:03:52 on dist-test-slave-dmhs"
server_key: "7257b6a8de0f0ec36694bbc5ede5b9a7"
server_key_iv: "fd7953b8f206e167efe90c1aa9d0cc65"
server_key_version: "encryptionkey@0"
I20260928 15:03:52.114845 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "aee9bae3b8654923a54d64ccc0a02b70"
format_stamp: "Formatted at 2026-09-28 15:03:52 on dist-test-slave-dmhs"
server_key: "7257b6a8de0f0ec36694bbc5ede5b9a7"
server_key_iv: "fd7953b8f206e167efe90c1aa9d0cc65"
server_key_version: "encryptionkey@0"
I20260928 15:03:52.119664 18801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 15:03:52.122780 18815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:52.123926 18801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:52.124075 18801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "aee9bae3b8654923a54d64ccc0a02b70"
format_stamp: "Formatted at 2026-09-28 15:03:52 on dist-test-slave-dmhs"
server_key: "7257b6a8de0f0ec36694bbc5ede5b9a7"
server_key_iv: "fd7953b8f206e167efe90c1aa9d0cc65"
server_key_version: "encryptionkey@0"
I20260928 15:03:52.124266 18801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:52.141242 18801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:52.141986 18801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:52.142258 18801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:52.143141 18801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:52.144482 18801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:52.144606 18801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:52.144724 18801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:52.144807 18801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:52.175870 18801 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:37913
I20260928 15:03:52.175925 18927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:37913 every 8 connection(s)
I20260928 15:03:52.177135 18801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:52.179898 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18801
I20260928 15:03:52.180059 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:52.180701 9267 external_mini_cluster.cc:1428] Setting key 587d9c82f42524e94cbe91efc7cf938d
I20260928 15:03:52.183355 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41459
--builtin_ntp_servers=127.9.12.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 15:03:52.189576 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.179652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:43721 (local address 127.9.12.254:41459)
0928 15:03:52.179818 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:52.179822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:52.181425 (+ 1603us) server_negotiation.cc:407] Connection header received
0928 15:03:52.183440 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:52.183445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:52.183523 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:52.183630 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:52.184688 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.185356 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.186839 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.187271 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.187407 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.188771 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:52.188775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:52.188779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:52.188795 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:52.188826 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:52.188837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:52.188927 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:52.189212 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 15:03:52.189294 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:03:52.190850 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.180163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:41459 (local address 127.9.12.194:43721)
0928 15:03:52.181231 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:52.181287 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:52.183189 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:52.183851 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:52.183859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:52.183940 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:52.184535 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:52.184553 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.185505 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.185511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:52.186694 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:52.186705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.187499 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.187506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:52.187728 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.188459 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:52.188462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:52.188508 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:52.188584 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:52.188925 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:52.188948 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:52.188958 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:03:52.189147 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 15:03:52.189575 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":300,"threads_started":1}
I20260928 15:03:52.192203 18928 heartbeater.cc:347] Connected to a master server at 127.9.12.254:41459
I20260928 15:03:52.192545 18928 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:52.193415 18928 heartbeater.cc:511] Master 127.9.12.254:41459 requested a full tablet report, sending...
I20260928 15:03:52.194859 18611 ts_manager.cc:194] Registered new tserver with Master: aee9bae3b8654923a54d64ccc0a02b70 (127.9.12.194:37913)
I20260928 15:03:52.196067 18611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.194:43721
W20260928 15:03:52.507831 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:52.508081 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:52.508162 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:52.515051 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:52.515141 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:52.515249 18931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:52.523880 18931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:41459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:52.525079 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:52.526777 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:52.539490 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:52.539448 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:52.539686 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:52.540658 18931 server_base.cc:1034] running on GCE node
I20260928 15:03:52.541328 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:52.542208 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:52.543437 18931 hybrid_clock.cc:648] HybridClock initialized: now 1790607832543395 us; error 55 us; skew 500 ppm
I20260928 15:03:52.543744 18931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:52.545876 18931 webserver.cc:460] Webserver started at http://127.9.12.195:40621/ using document root <none> and password file <none>
I20260928 15:03:52.546533 18931 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:52.546679 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:52.547009 18931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:52.549754 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "211839e065e243c796286ff23f492d5b"
format_stamp: "Formatted at 2026-09-28 15:03:52 on dist-test-slave-dmhs"
server_key: "6609ca42fa8200928157991cf6c9d7ca"
server_key_iv: "fb8c99819e590f53338433bdc9e87524"
server_key_version: "encryptionkey@0"
I20260928 15:03:52.550523 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "211839e065e243c796286ff23f492d5b"
format_stamp: "Formatted at 2026-09-28 15:03:52 on dist-test-slave-dmhs"
server_key: "6609ca42fa8200928157991cf6c9d7ca"
server_key_iv: "fb8c99819e590f53338433bdc9e87524"
server_key_version: "encryptionkey@0"
I20260928 15:03:52.555382 18931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 15:03:52.558624 18945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:52.559806 18931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:52.559979 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "211839e065e243c796286ff23f492d5b"
format_stamp: "Formatted at 2026-09-28 15:03:52 on dist-test-slave-dmhs"
server_key: "6609ca42fa8200928157991cf6c9d7ca"
server_key_iv: "fb8c99819e590f53338433bdc9e87524"
server_key_version: "encryptionkey@0"
I20260928 15:03:52.560190 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:52.589185 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:52.590067 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:52.590370 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:52.591305 18931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:52.592753 18931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:52.592875 18931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:52.593010 18931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:52.593106 18931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:52.625869 18931 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:35833
I20260928 15:03:52.625926 19057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:35833 every 8 connection(s)
I20260928 15:03:52.627225 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:52.627758 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 18931
I20260928 15:03:52.627946 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:52.628629 9267 external_mini_cluster.cc:1428] Setting key 4c23e068d0a82ab8ab7db336dce3fde0
I20260928 15:03:52.639985 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.629960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:45069 (local address 127.9.12.254:41459)
0928 15:03:52.630184 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:52.630192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:52.631809 (+ 1617us) server_negotiation.cc:407] Connection header received
0928 15:03:52.633684 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:52.633689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:52.633783 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:52.633891 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:52.634946 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.635687 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.637302 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.637704 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.637842 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.639181 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:52.639187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:52.639192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:52.639217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:52.639251 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:52.639262 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:52.639365 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:52.639584 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 15:03:52.639675 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:03:52.640985 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.630646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:41459 (local address 127.9.12.195:45069)
0928 15:03:52.631642 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:52.631697 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:52.633420 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:52.634133 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:52.634140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:52.634225 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:52.634777 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:52.634793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.635847 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.635857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:52.637156 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:52.637167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.637925 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.637932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:52.638165 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.638860 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:52.638863 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:52.638899 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:52.638970 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:52.639365 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:52.639377 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:52.639386 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:03:52.639572 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 15:03:52.639927 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":333,"threads_started":1}
I20260928 15:03:52.642220 19058 heartbeater.cc:347] Connected to a master server at 127.9.12.254:41459
I20260928 15:03:52.642509 19058 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:52.643314 19058 heartbeater.cc:511] Master 127.9.12.254:41459 requested a full tablet report, sending...
I20260928 15:03:52.644981 18611 ts_manager.cc:194] Registered new tserver with Master: 211839e065e243c796286ff23f492d5b (127.9.12.195:35833)
I20260928 15:03:52.645720 18611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.195:45069
I20260928 15:03:52.656150 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:03:52.661971 18799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:52.660575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.9.12.254:41459)
0928 15:03:52.660778 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:52.660787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:52.661170 (+ 383us) server_negotiation.cc:407] Connection header received
0928 15:03:52.661295 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:52.661299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:52.661370 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:52.661448 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:52.661737 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38178: BlockingRecv error: recv got EOF from 127.0.0.1:38178 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:03:52.662707 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18670
I20260928 15:03:52.670558 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.663564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.9.12.193:38723)
0928 15:03:52.664060 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:52.664067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:52.664135 (+ 68us) server_negotiation.cc:407] Connection header received
0928 15:03:52.664205 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:52.664211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:52.664543 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:52.664688 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:52.665044 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.665995 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.667372 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.667943 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.668077 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.669180 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:52.669190 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:52.669212 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:52.669751 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:52.669793 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:52.669811 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:52.669935 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:52.670087 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 15:03:52.670229 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260928 15:03:52.671656 18792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:52.751331 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18670
I20260928 15:03:52.765616 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18801
I20260928 15:03:52.773319 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.766525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.9.12.194:37913)
0928 15:03:52.767160 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:52.767167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:52.767194 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:03:52.767255 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:52.767262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:52.767533 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:52.767664 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:52.768049 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.769136 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.770348 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.770881 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.770977 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.771884 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:52.771892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:52.771904 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:52.772597 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:52.772643 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:52.772663 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:52.772751 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:52.772889 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 15:03:52.773011 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":323,"threads_started":1}
I20260928 15:03:52.774497 18923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:52.854030 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18801
I20260928 15:03:52.870016 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18931
I20260928 15:03:52.878110 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:52.871015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.9.12.195:35833)
0928 15:03:52.871647 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:52.871654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:52.871684 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:03:52.871747 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:52.871753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:52.871957 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:52.872096 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:52.872551 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.873705 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.874947 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:52.875497 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:52.875604 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:52.876728 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:52.876735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:52.876747 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:52.877322 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:52.877364 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:52.877390 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:52.877473 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:52.877591 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 15:03:52.877729 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":324,"threads_started":1}
I20260928 15:03:52.879281 19053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:52.959375 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18931
I20260928 15:03:52.974970 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 18580
I20260928 15:03:52.976083 18641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:53.074885 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 18580
2026-09-28T15:03:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2333 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T15:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:53Z Disabled control of system clock
I20260928 15:03:53.145892 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45325
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:03:53.463847 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:53.464154 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:53.464236 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:53.472558 19077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:53.472656 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:53.472702 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:53.472760 19077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:53.472802 19077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:53.481884 19077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:45325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:53.483067 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:53.484848 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:53.496773 19082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:53.496758 19083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:53.496942 19085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:53.497537 19077 server_base.cc:1034] running on GCE node
I20260928 15:03:53.498301 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:53.499637 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:53.500869 19077 hybrid_clock.cc:648] HybridClock initialized: now 1790607833500810 us; error 49 us; skew 500 ppm
I20260928 15:03:53.501224 19077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:53.503512 19077 webserver.cc:460] Webserver started at http://127.9.12.254:33097/ using document root <none> and password file <none>
I20260928 15:03:53.504216 19077 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:53.504369 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:53.504691 19077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:53.507535 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "dac071eaea244fe586435fb6eb7cded4"
format_stamp: "Formatted at 2026-09-28 15:03:53 on dist-test-slave-dmhs"
server_key: "b161fbf7df322d10c7ef19ac743f7971"
server_key_iv: "475b74472c39a85e5439d42285dd97bb"
server_key_version: "encryptionkey@0"
I20260928 15:03:53.508387 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "dac071eaea244fe586435fb6eb7cded4"
format_stamp: "Formatted at 2026-09-28 15:03:53 on dist-test-slave-dmhs"
server_key: "b161fbf7df322d10c7ef19ac743f7971"
server_key_iv: "475b74472c39a85e5439d42285dd97bb"
server_key_version: "encryptionkey@0"
I20260928 15:03:53.513268 19077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:03:53.516738 19091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:53.517995 19077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:03:53.518174 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "dac071eaea244fe586435fb6eb7cded4"
format_stamp: "Formatted at 2026-09-28 15:03:53 on dist-test-slave-dmhs"
server_key: "b161fbf7df322d10c7ef19ac743f7971"
server_key_iv: "475b74472c39a85e5439d42285dd97bb"
server_key_version: "encryptionkey@0"
I20260928 15:03:53.518396 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:53.524154 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:53.524962 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:53.525272 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:53.542899 19077 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:45325
I20260928 15:03:53.542932 19142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:45325 every 8 connection(s)
I20260928 15:03:53.544231 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:53.546579 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19077
I20260928 15:03:53.546805 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:53.547642 9267 external_mini_cluster.cc:1428] Setting key 9b4bd1ddf518073aedc533865e15535b
I20260928 15:03:53.548811 19143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:53.556885 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:53.549786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.9.12.254:45325)
0928 15:03:53.550775 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:53.550786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:53.550827 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:03:53.553224 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:53.553247 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:53.553686 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:53.554020 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:53.555175 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:53.555181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:53.555232 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:53.555260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:53.555328 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:53.555346 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:53.555528 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:53.555632 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 15:03:53.555926 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":382,"threads_started":1}
I20260928 15:03:53.562108 19143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4: Bootstrap starting.
I20260928 15:03:53.565232 19143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:53.566237 19143 log.cc:824] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:53.568964 19143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4: No bootstrap required, opened a new log
I20260928 15:03:53.576458 19143 raft_consensus.cc:348] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac071eaea244fe586435fb6eb7cded4" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45325 } }
I20260928 15:03:53.576834 19143 raft_consensus.cc:374] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:53.576937 19143 raft_consensus.cc:729] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac071eaea244fe586435fb6eb7cded4, State: Initialized, Role: FOLLOWER
I20260928 15:03:53.577571 19143 consensus_queue.cc:260] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac071eaea244fe586435fb6eb7cded4" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45325 } }
I20260928 15:03:53.577785 19143 raft_consensus.cc:388] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:53.577977 19143 raft_consensus.cc:482] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:53.578183 19143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:53.580531 19143 raft_consensus.cc:504] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac071eaea244fe586435fb6eb7cded4" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45325 } }
I20260928 15:03:53.581072 19143 leader_election.cc:302] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dac071eaea244fe586435fb6eb7cded4; no voters:
I20260928 15:03:53.581663 19143 leader_election.cc:288] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:53.581826 19148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:53.582609 19148 raft_consensus.cc:686] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [term 1 LEADER]: Becoming Leader. State: Replica: dac071eaea244fe586435fb6eb7cded4, State: Running, Role: LEADER
I20260928 15:03:53.583180 19148 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac071eaea244fe586435fb6eb7cded4" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45325 } }
I20260928 15:03:53.583877 19143 sys_catalog.cc:564] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:53.586050 19150 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac071eaea244fe586435fb6eb7cded4. Latest consensus state: current_term: 1 leader_uuid: "dac071eaea244fe586435fb6eb7cded4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac071eaea244fe586435fb6eb7cded4" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45325 } } }
I20260928 15:03:53.586401 19150 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:53.587440 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dac071eaea244fe586435fb6eb7cded4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac071eaea244fe586435fb6eb7cded4" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45325 } } }
I20260928 15:03:53.587868 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:53.588660 19154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:53.594349 19154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:53.595099 19165 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:03:53.605012 19154 catalog_manager.cc:1347] Generated new cluster ID: 43865d49e136472192f691476d672921
I20260928 15:03:53.605217 19154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:53.619009 19154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:53.619503 19154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:53.625969 19154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dac071eaea244fe586435fb6eb7cded4: Generated new TSK 0
I20260928 15:03:53.626766 19154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:53.640597 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45325
--builtin_ntp_servers=127.9.12.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:03:53.966388 19167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:53.966671 19167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:53.966841 19167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:53.974512 19167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:53.974620 19167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:53.974838 19167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:53.984004 19167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:53.985428 19167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:53.987248 19167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:54.000711 19172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:54.000684 19173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:54.000697 19175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:54.001621 19167 server_base.cc:1034] running on GCE node
I20260928 15:03:54.002251 19167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:54.003182 19167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:54.004441 19167 hybrid_clock.cc:648] HybridClock initialized: now 1790607834004367 us; error 49 us; skew 500 ppm
I20260928 15:03:54.004762 19167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:54.007020 19167 webserver.cc:460] Webserver started at http://127.9.12.193:44071/ using document root <none> and password file <none>
I20260928 15:03:54.007686 19167 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:54.007826 19167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:54.008168 19167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:54.010900 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "1000f751c998488cad3f43fb94cd2e57"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "9167f9c4bd337435fee5d2eeba055c64"
server_key_iv: "f6c57a3aa5304e0d3fd1c53c5b4b0e75"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.011693 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "1000f751c998488cad3f43fb94cd2e57"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "9167f9c4bd337435fee5d2eeba055c64"
server_key_iv: "f6c57a3aa5304e0d3fd1c53c5b4b0e75"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.016502 19167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 15:03:54.019600 19181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.020867 19167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:54.021054 19167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "1000f751c998488cad3f43fb94cd2e57"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "9167f9c4bd337435fee5d2eeba055c64"
server_key_iv: "f6c57a3aa5304e0d3fd1c53c5b4b0e75"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.021255 19167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:54.027073 19167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:54.027834 19167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:54.028115 19167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:54.029115 19167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:54.030440 19167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:54.030567 19167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.030690 19167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:54.030792 19167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.063717 19167 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:40655
I20260928 15:03:54.063769 19293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:40655 every 8 connection(s)
I20260928 15:03:54.065127 19167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:54.072753 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19167
I20260928 15:03:54.072956 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:54.073025 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.067640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51005 (local address 127.9.12.254:45325)
0928 15:03:54.068085 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:54.068089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:54.069208 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 15:03:54.070861 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:54.070866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:54.070941 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:54.071072 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:54.072195 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:54.072201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:54.072205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:54.072221 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:54.072318 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:54.072337 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:54.072460 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:54.072663 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 15:03:54.072730 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":250,"threads_started":1}
I20260928 15:03:54.073598 9267 external_mini_cluster.cc:1428] Setting key bb4dd3ee97195e1fd4cff8c4902f764e
I20260928 15:03:54.073833 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.068155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45325 (local address 127.9.12.193:51005)
0928 15:03:54.069067 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:54.069101 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:54.070585 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:54.071283 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:54.071291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:54.071372 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:54.071930 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:54.071932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:54.071964 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:54.072038 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:54.072456 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:54.072464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:54.072478 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:03:54.072684 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 15:03:54.072883 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":273,"threads_started":1}
I20260928 15:03:54.075277 19294 heartbeater.cc:347] Connected to a master server at 127.9.12.254:45325
I20260928 15:03:54.075562 19294 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:54.076228 19294 heartbeater.cc:511] Master 127.9.12.254:45325 requested a full tablet report, sending...
I20260928 15:03:54.076746 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45325
--builtin_ntp_servers=127.9.12.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:03:54.078284 19107 ts_manager.cc:194] Registered new tserver with Master: 1000f751c998488cad3f43fb94cd2e57 (127.9.12.193:40655)
W20260928 15:03:54.395534 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:54.395826 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:54.395983 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:54.403141 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:54.403272 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:54.403450 19298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:54.412485 19298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:54.413692 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:54.415386 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:54.428289 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:54.428340 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:54.428366 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:54.429302 19298 server_base.cc:1034] running on GCE node
I20260928 15:03:54.429800 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:54.430619 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:54.431813 19298 hybrid_clock.cc:648] HybridClock initialized: now 1790607834431779 us; error 22 us; skew 500 ppm
I20260928 15:03:54.432076 19298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:54.434204 19298 webserver.cc:460] Webserver started at http://127.9.12.194:41293/ using document root <none> and password file <none>
I20260928 15:03:54.434825 19298 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:54.434906 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:54.435171 19298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:54.437992 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "444ad5e1173749a19503633996c14443"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "47465492e8746f89edd4fd7d5c07418a"
server_key_iv: "573bb562b854db2daa3ce918b19d1973"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.438745 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "444ad5e1173749a19503633996c14443"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "47465492e8746f89edd4fd7d5c07418a"
server_key_iv: "573bb562b854db2daa3ce918b19d1973"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.443395 19298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 15:03:54.448441 19312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.450084 19298 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:03:54.450237 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "444ad5e1173749a19503633996c14443"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "47465492e8746f89edd4fd7d5c07418a"
server_key_iv: "573bb562b854db2daa3ce918b19d1973"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.450491 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:54.457326 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:54.458158 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:54.458415 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:54.459283 19298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:54.460640 19298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:54.460763 19298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.460889 19298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:54.460978 19298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.493068 19298 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:37801
I20260928 15:03:54.493090 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:37801 every 8 connection(s)
I20260928 15:03:54.494340 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:54.498802 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19298
I20260928 15:03:54.499009 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:54.499455 9267 external_mini_cluster.cc:1428] Setting key 6d6c7eb8c25e45a3c7fed757762d6ba0
I20260928 15:03:54.502246 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45325
--builtin_ntp_servers=127.9.12.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:03:54.502503 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.496779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:40917 (local address 127.9.12.254:45325)
0928 15:03:54.496985 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:54.496991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:54.498347 (+ 1356us) server_negotiation.cc:407] Connection header received
0928 15:03:54.500183 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:54.500190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:54.500304 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:54.500408 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:54.501760 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:54.501766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:54.501770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:54.501786 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:54.501824 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:54.501835 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:54.501944 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:54.502165 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 15:03:54.502229 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:03:54.503328 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.497337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45325 (local address 127.9.12.194:40917)
0928 15:03:54.498206 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:54.498240 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:54.499848 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:54.500656 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:54.500666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:54.500774 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:54.501483 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:54.501486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:54.501533 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:54.501608 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:54.501921 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:54.501930 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:54.501937 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:03:54.502221 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 15:03:54.502445 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":257,"threads_started":1}
I20260928 15:03:54.504690 19425 heartbeater.cc:347] Connected to a master server at 127.9.12.254:45325
I20260928 15:03:54.504980 19425 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:54.505601 19425 heartbeater.cc:511] Master 127.9.12.254:45325 requested a full tablet report, sending...
I20260928 15:03:54.507117 19107 ts_manager.cc:194] Registered new tserver with Master: 444ad5e1173749a19503633996c14443 (127.9.12.194:37801)
W20260928 15:03:54.819032 19428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:54.819357 19428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:54.819532 19428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:54.827147 19428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:54.827276 19428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:54.827451 19428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:54.837193 19428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:54.838486 19428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:54.840204 19428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:54.853447 19436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:54.853463 19434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:54.853507 19433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:54.857617 19428 server_base.cc:1034] running on GCE node
I20260928 15:03:54.858264 19428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:54.859165 19428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:54.860363 19428 hybrid_clock.cc:648] HybridClock initialized: now 1790607834860304 us; error 64 us; skew 500 ppm
I20260928 15:03:54.860685 19428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:03:54.862960 19428 webserver.cc:460] Webserver started at http://127.9.12.195:33111/ using document root <none> and password file <none>
I20260928 15:03:54.863670 19428 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:54.863790 19428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:54.864188 19428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:54.867038 19428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "18324ce62549457989d4ff588e47be37"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "fa06d9d2c847ba33ff53363e1d2c4268"
server_key_iv: "c08aa2dc279959ef9bb2e6696e285db2"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.867834 19428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "18324ce62549457989d4ff588e47be37"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "fa06d9d2c847ba33ff53363e1d2c4268"
server_key_iv: "c08aa2dc279959ef9bb2e6696e285db2"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.872896 19428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:03:54.876323 19442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.877534 19428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:03:54.877710 19428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "18324ce62549457989d4ff588e47be37"
format_stamp: "Formatted at 2026-09-28 15:03:54 on dist-test-slave-dmhs"
server_key: "fa06d9d2c847ba33ff53363e1d2c4268"
server_key_iv: "c08aa2dc279959ef9bb2e6696e285db2"
server_key_version: "encryptionkey@0"
I20260928 15:03:54.877918 19428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:54.883432 19428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:54.884222 19428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:54.884547 19428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:54.885463 19428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:54.886765 19428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:54.886897 19428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.887063 19428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:54.887152 19428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:54.919425 19428 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:40609
I20260928 15:03:54.919483 19554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:40609 every 8 connection(s)
I20260928 15:03:54.920845 19428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:54.925379 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19428
I20260928 15:03:54.925597 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:54.926030 9267 external_mini_cluster.cc:1428] Setting key d02cf3f8e26d9019d5791c1437066842
I20260928 15:03:54.929155 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.923343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:38605 (local address 127.9.12.254:45325)
0928 15:03:54.923504 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:54.923509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:54.924935 (+ 1426us) server_negotiation.cc:407] Connection header received
0928 15:03:54.926762 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:54.926768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:54.926848 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:54.926993 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:54.928402 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:54.928408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:54.928414 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:54.928434 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:54.928466 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:54.928486 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:54.928618 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:54.928809 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 15:03:54.928877 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:03:54.930037 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.923805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:45325 (local address 127.9.12.195:38605)
0928 15:03:54.924786 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:54.924820 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:54.926444 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:54.927190 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:54.927199 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:54.927334 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:54.928076 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:54.928078 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:54.928114 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:54.928197 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:54.928582 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:54.928593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:54.928603 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:03:54.928891 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 15:03:54.929149 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":261,"threads_started":1}
I20260928 15:03:54.931221 19555 heartbeater.cc:347] Connected to a master server at 127.9.12.254:45325
I20260928 15:03:54.931443 19555 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:54.931933 19555 heartbeater.cc:511] Master 127.9.12.254:45325 requested a full tablet report, sending...
I20260928 15:03:54.933236 19107 ts_manager.cc:194] Registered new tserver with Master: 18324ce62549457989d4ff588e47be37 (127.9.12.195:40609)
I20260928 15:03:54.941133 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:03:54.947039 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:54.945490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.9.12.254:45325)
0928 15:03:54.945698 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:54.945706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:54.946102 (+ 396us) server_negotiation.cc:407] Connection header received
0928 15:03:54.946300 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:54.946304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:54.946356 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:54.946450 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:54.946819 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39492: BlockingRecv error: recv got EOF from 127.0.0.1:39492 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:03:54.947932 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19167
I20260928 15:03:54.951242 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:54.948921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.9.12.193:40655)
0928 15:03:54.949395 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:54.949402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:54.949425 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:03:54.949478 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:54.949483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:54.949666 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:54.949781 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:54.950602 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:54.950608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:54.950618 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:54.950635 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:54.950670 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:54.950682 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:54.950780 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:54.950856 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:03:54.950966 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
I20260928 15:03:54.952373 19289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:55.031629 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19167
I20260928 15:03:55.047010 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19298
I20260928 15:03:55.050450 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:55.047921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.9.12.194:37801)
0928 15:03:55.048501 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:55.048507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:55.048533 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:03:55.048593 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:55.048598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:55.048802 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:55.048937 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:55.049826 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:55.049833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:55.049843 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:55.049856 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:55.049907 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:55.049921 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:55.050034 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:55.050127 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 15:03:55.050213 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260928 15:03:55.051638 19420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:55.130539 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19298
I20260928 15:03:55.145725 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19428
I20260928 15:03:55.149344 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:55.146669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.9.12.195:40609)
0928 15:03:55.147322 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:55.147328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:55.147359 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:03:55.147421 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:55.147426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:55.147643 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:55.147762 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:55.148728 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:55.148734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:55.148745 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:55.148761 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:55.148799 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:55.148813 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:55.148940 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:55.149007 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:03:55.149084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":357,"threads_started":1}
I20260928 15:03:55.150586 19550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:55.231212 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19428
I20260928 15:03:55.246795 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19077
I20260928 15:03:55.247905 19138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:55.347381 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19077
2026-09-28T15:03:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2273 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:55 dist-test-slave-dmhs krb5kdc[19573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:55 dist-test-slave-dmhs krb5kdc[19573](info): set up 2 sockets
Sep 28 15:03:55 dist-test-slave-dmhs krb5kdc[19573](info): commencing operation
krb5kdc: starting...
W20260928 15:03:56.429766 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:03:56 dist-test-slave-dmhs krb5kdc[19573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:03:56.652213 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37549
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:37549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:56.968912 19586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:56.969182 19586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:56.969329 19586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:56.976259 19586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:56.976364 19586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:56.976408 19586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:56.976517 19586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:56.976598 19586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:56.986256 19586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:37549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:56.987540 19586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:56.988883 19586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:57.000388 19591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:57.000494 19594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:57.000469 19592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:57.001545 19586 server_base.cc:1034] running on GCE node
I20260928 15:03:57.002270 19586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:57.003674 19586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:57.004905 19586 hybrid_clock.cc:648] HybridClock initialized: now 1790607837004877 us; error 34 us; skew 500 ppm
I20260928 15:03:57.005256 19586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:57 dist-test-slave-dmhs krb5kdc[19573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:57.008328 19586 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:03:57.009680 19586 webserver.cc:460] Webserver started at http://127.9.12.254:38503/ using document root <none> and password file <none>
I20260928 15:03:57.010353 19586 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:57.010530 19586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:57.010818 19586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:57.013617 19586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "cd21fab45b894ca7aca070ea227660a2"
format_stamp: "Formatted at 2026-09-28 15:03:57 on dist-test-slave-dmhs"
server_key: "f0a76c587fff124b87ead458258de4bb"
server_key_iv: "8ef08d3243850ebb8b89ba94c27ce038"
server_key_version: "encryptionkey@0"
I20260928 15:03:57.014371 19586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "cd21fab45b894ca7aca070ea227660a2"
format_stamp: "Formatted at 2026-09-28 15:03:57 on dist-test-slave-dmhs"
server_key: "f0a76c587fff124b87ead458258de4bb"
server_key_iv: "8ef08d3243850ebb8b89ba94c27ce038"
server_key_version: "encryptionkey@0"
I20260928 15:03:57.019273 19586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:03:57.022535 19601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:57.023828 19586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:03:57.024003 19586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "cd21fab45b894ca7aca070ea227660a2"
format_stamp: "Formatted at 2026-09-28 15:03:57 on dist-test-slave-dmhs"
server_key: "f0a76c587fff124b87ead458258de4bb"
server_key_iv: "8ef08d3243850ebb8b89ba94c27ce038"
server_key_version: "encryptionkey@0"
I20260928 15:03:57.024191 19586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:57.049396 19586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:57.053452 19586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:57.053722 19586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:57.071408 19586 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:37549
I20260928 15:03:57.071439 19652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:37549 every 8 connection(s)
I20260928 15:03:57.072687 19586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:57.074185 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19586
I20260928 15:03:57.074411 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:03:57.075197 9267 external_mini_cluster.cc:1428] Setting key da8d467255d53861adc0fe720fa7ce91
I20260928 15:03:57.077167 19653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:57.087527 19653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2: Bootstrap starting.
Sep 28 15:03:57 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:57.090946 19653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:57.091954 19653 log.cc:824] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:57.095245 19653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2: No bootstrap required, opened a new log
I20260928 15:03:57.097489 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:57.077180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.9.12.254:37549)
0928 15:03:57.077887 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:57.077898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:57.077935 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:03:57.079518 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:57.079538 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:57.079893 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:57.080308 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:57.081697 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:57.082962 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:57.084521 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:57.085217 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:57.085471 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:57.089833 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:57.089840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:57.089853 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:57.089885 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:57.093449 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:57.094214 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:57.094223 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:57.094229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:57.094384 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:57.094802 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:57.094805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:57.094807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:57.095340 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:57.095678 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:57.096112 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 15:03:57.096546 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":318,"threads_started":1}
I20260928 15:03:57.103327 19653 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21fab45b894ca7aca070ea227660a2" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37549 } }
I20260928 15:03:57.103730 19653 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:57.103847 19653 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21fab45b894ca7aca070ea227660a2, State: Initialized, Role: FOLLOWER
I20260928 15:03:57.104432 19653 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21fab45b894ca7aca070ea227660a2" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37549 } }
I20260928 15:03:57.104712 19653 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:57.104887 19653 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:57.105095 19653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:57.107252 19653 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21fab45b894ca7aca070ea227660a2" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37549 } }
I20260928 15:03:57.107673 19653 leader_election.cc:302] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd21fab45b894ca7aca070ea227660a2; no voters:
I20260928 15:03:57.108198 19653 leader_election.cc:288] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:57.108378 19658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:57.109130 19658 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [term 1 LEADER]: Becoming Leader. State: Replica: cd21fab45b894ca7aca070ea227660a2, State: Running, Role: LEADER
I20260928 15:03:57.109724 19658 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21fab45b894ca7aca070ea227660a2" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37549 } }
I20260928 15:03:57.110306 19653 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:57.112593 19660 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd21fab45b894ca7aca070ea227660a2. Latest consensus state: current_term: 1 leader_uuid: "cd21fab45b894ca7aca070ea227660a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21fab45b894ca7aca070ea227660a2" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37549 } } }
I20260928 15:03:57.112946 19660 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:57.113405 19659 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd21fab45b894ca7aca070ea227660a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd21fab45b894ca7aca070ea227660a2" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37549 } } }
I20260928 15:03:57.113727 19659 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:57.121795 19664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:57.127514 19664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:57.136477 19664 catalog_manager.cc:1347] Generated new cluster ID: d77021c520274591ac7a8efc4c478153
I20260928 15:03:57.136590 19664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:57.152321 19664 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:57.153609 19664 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:57.160666 19664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2: Generated new TSK 0
I20260928 15:03:57.161450 19664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:03:57.231848 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37549
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:57.550323 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:57.550578 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:57.550701 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:57.557605 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:57.557709 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:57.557924 19679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:03:57.567111 19679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:57.568365 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:57.570097 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:57.582866 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:57.582839 19685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:57.582865 19687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:57.583675 19679 server_base.cc:1034] running on GCE node
I20260928 15:03:57.584319 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:57.585220 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:57.586448 19679 hybrid_clock.cc:648] HybridClock initialized: now 1790607837586405 us; error 55 us; skew 500 ppm
I20260928 15:03:57.586763 19679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:57 dist-test-slave-dmhs krb5kdc[19573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:57.589908 19679 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:03:57.591424 19679 webserver.cc:460] Webserver started at http://127.9.12.193:33373/ using document root <none> and password file <none>
I20260928 15:03:57.592080 19679 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:57.592224 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:57.592561 19679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:57.595382 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "2891d07f75854c8ebe79e49880dcee57"
format_stamp: "Formatted at 2026-09-28 15:03:57 on dist-test-slave-dmhs"
server_key: "64af3dd827006d244b805f18bd56d088"
server_key_iv: "487486d5657c1f43941f5911ffff1cd9"
server_key_version: "encryptionkey@0"
I20260928 15:03:57.596145 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "2891d07f75854c8ebe79e49880dcee57"
format_stamp: "Formatted at 2026-09-28 15:03:57 on dist-test-slave-dmhs"
server_key: "64af3dd827006d244b805f18bd56d088"
server_key_iv: "487486d5657c1f43941f5911ffff1cd9"
server_key_version: "encryptionkey@0"
I20260928 15:03:57.601161 19679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:03:57.604496 19694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:57.605687 19679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:03:57.605855 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "2891d07f75854c8ebe79e49880dcee57"
format_stamp: "Formatted at 2026-09-28 15:03:57 on dist-test-slave-dmhs"
server_key: "64af3dd827006d244b805f18bd56d088"
server_key_iv: "487486d5657c1f43941f5911ffff1cd9"
server_key_version: "encryptionkey@0"
I20260928 15:03:57.606041 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:57.622815 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:57.626796 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:57.627113 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:57.628093 19679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:57.629558 19679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:57.629660 19679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:57.629796 19679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:57.629882 19679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:57.662249 19679 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:32881
I20260928 15:03:57.662297 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:32881 every 8 connection(s)
I20260928 15:03:57.663548 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:57.673025 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19679
I20260928 15:03:57.673237 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:57.673997 9267 external_mini_cluster.cc:1428] Setting key 4e8517f20d2a470e61aa7532977cfaa2
Sep 28 15:03:57 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:57.682674 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:57.666004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51551 (local address 127.9.12.254:37549)
0928 15:03:57.666484 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:57.666491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:57.667406 (+ 915us) server_negotiation.cc:407] Connection header received
0928 15:03:57.669172 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:57.669177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:57.669240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:57.669333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:57.670689 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:57.671389 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:57.672908 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:57.673543 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:57.673678 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:57.677207 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:57.677212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:57.677215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:57.677245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:57.679742 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:57.680467 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:57.680471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:57.680473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:57.680582 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:57.681030 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:57.681035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:57.681038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:57.681291 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:57.681441 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:57.682126 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 15:03:57.682335 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260928 15:03:57.683537 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:57.666549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37549 (local address 127.9.12.193:51551)
0928 15:03:57.667249 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:57.667296 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:57.668932 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:57.669528 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:57.669536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:57.670003 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:57.670546 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:57.670562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:57.671512 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:57.671516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:57.672764 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:57.672776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:57.673732 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:57.673737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:57.673917 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:57.674600 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:57.674620 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:57.676997 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:57.679878 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:57.679886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:57.679900 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:57.680339 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:57.680708 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:57.680713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:57.680717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:57.680905 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:57.681415 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:57.681422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:57.681734 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 15:03:57.682015 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 15:03:57.682502 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
I20260928 15:03:57.685072 19807 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37549
I20260928 15:03:57.685379 19807 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:57.686206 19807 heartbeater.cc:511] Master 127.9.12.254:37549 requested a full tablet report, sending...
I20260928 15:03:57.688578 19618 ts_manager.cc:194] Registered new tserver with Master: 2891d07f75854c8ebe79e49880dcee57 (127.9.12.193:32881)
I20260928 15:03:57.690701 19618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:51551
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:03:57.729374 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37549
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:58.042899 19813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:58.043167 19813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:58.043260 19813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:58.050176 19813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:58.050284 19813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:58.050403 19813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:03:58.060354 19813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:58.061570 19813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:58.063061 19813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:58.075794 19818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:58.075956 19819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:58.076377 19821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:58.076722 19813 server_base.cc:1034] running on GCE node
I20260928 15:03:58.077319 19813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:58.078194 19813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:58.079418 19813 hybrid_clock.cc:648] HybridClock initialized: now 1790607838079366 us; error 43 us; skew 500 ppm
I20260928 15:03:58.079732 19813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:58 dist-test-slave-dmhs krb5kdc[19573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607838, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:58.082998 19813 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:03:58.084496 19813 webserver.cc:460] Webserver started at http://127.9.12.194:33205/ using document root <none> and password file <none>
I20260928 15:03:58.085153 19813 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:58.085295 19813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:58.085582 19813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:58.088491 19813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "bd5f285e19b0450eb49073d1506ca0ed"
format_stamp: "Formatted at 2026-09-28 15:03:58 on dist-test-slave-dmhs"
server_key: "5e277c71f4b28845e637cf9b4d42ad9a"
server_key_iv: "441ee601827a34952c69d557fc4d4db8"
server_key_version: "encryptionkey@0"
I20260928 15:03:58.089264 19813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5f285e19b0450eb49073d1506ca0ed"
format_stamp: "Formatted at 2026-09-28 15:03:58 on dist-test-slave-dmhs"
server_key: "5e277c71f4b28845e637cf9b4d42ad9a"
server_key_iv: "441ee601827a34952c69d557fc4d4db8"
server_key_version: "encryptionkey@0"
I20260928 15:03:58.094209 19813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:03:58.097483 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:58.098673 19813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:58.098845 19813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "bd5f285e19b0450eb49073d1506ca0ed"
format_stamp: "Formatted at 2026-09-28 15:03:58 on dist-test-slave-dmhs"
server_key: "5e277c71f4b28845e637cf9b4d42ad9a"
server_key_iv: "441ee601827a34952c69d557fc4d4db8"
server_key_version: "encryptionkey@0"
I20260928 15:03:58.099032 19813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:58.118950 19813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:58.122722 19813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:58.123023 19813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:58.124156 19813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:58.125370 19813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:58.125474 19813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:58.125612 19813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:58.125681 19813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:58.158211 19813 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:39945
I20260928 15:03:58.158267 19940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:39945 every 8 connection(s)
I20260928 15:03:58.159462 19813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:58.161124 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19813
I20260928 15:03:58.161319 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:58.161716 9267 external_mini_cluster.cc:1428] Setting key 740d565bde98a26fcc1de5b1676887b0
Sep 28 15:03:58 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607838, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:58.180234 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.162070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:40619 (local address 127.9.12.254:37549)
0928 15:03:58.162274 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.162281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.163589 (+ 1308us) server_negotiation.cc:407] Connection header received
0928 15:03:58.165989 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.165997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.166099 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.166200 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:58.167628 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.168784 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.170273 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.171038 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.171182 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.174807 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:58.174812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:58.174815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:58.174845 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:58.177297 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.178014 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.178019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.178024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.178158 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.178607 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.178613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.178616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.178868 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:58.179022 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.179655 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.179871 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:03:58.181246 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.162610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37549 (local address 127.9.12.194:40619)
0928 15:03:58.163405 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.163460 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.165633 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.166394 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.166403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.166896 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:58.167455 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.167471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.168915 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.168919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.170129 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.170140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.171240 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.171244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.171433 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.172103 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:58.172129 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:58.174587 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:58.177435 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:58.177443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:58.177458 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:58.177874 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:58.178267 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:58.178272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:58.178275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:58.178473 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:58.178994 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:58.179001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:58.179309 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.179575 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.180072 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260928 15:03:58.182595 19941 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37549
I20260928 15:03:58.182880 19941 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:58.183552 19941 heartbeater.cc:511] Master 127.9.12.254:37549 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:03:58.185071 19618 ts_manager.cc:194] Registered new tserver with Master: bd5f285e19b0450eb49073d1506ca0ed (127.9.12.194:39945)
I20260928 15:03:58.185855 19618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:40619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:03:58.215188 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37549
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:58.532675 19946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:58.532927 19946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:58.533030 19946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:58.539943 19946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:58.540056 19946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:58.540165 19946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:03:58.549398 19946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:03:58.550549 19946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:58.551774 19946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:58.564431 19952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:58.564504 19954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:58.564558 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:58.565402 19946 server_base.cc:1034] running on GCE node
I20260928 15:03:58.566033 19946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:58.566980 19946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:58.568199 19946 hybrid_clock.cc:648] HybridClock initialized: now 1790607838568153 us; error 59 us; skew 500 ppm
I20260928 15:03:58.568521 19946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:03:58 dist-test-slave-dmhs krb5kdc[19573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607838, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:58.571382 19946 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:03:58.572769 19946 webserver.cc:460] Webserver started at http://127.9.12.195:37975/ using document root <none> and password file <none>
I20260928 15:03:58.573380 19946 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:58.573521 19946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:58.573807 19946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:58.576629 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "305401fb6d4f4f1d95c3f055eb4ab549"
format_stamp: "Formatted at 2026-09-28 15:03:58 on dist-test-slave-dmhs"
server_key: "d8e30c3d7ae6236ff4692cee1e0b0dfe"
server_key_iv: "8b82d4848bea9b3d94a397b960b43f65"
server_key_version: "encryptionkey@0"
I20260928 15:03:58.577364 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "305401fb6d4f4f1d95c3f055eb4ab549"
format_stamp: "Formatted at 2026-09-28 15:03:58 on dist-test-slave-dmhs"
server_key: "d8e30c3d7ae6236ff4692cee1e0b0dfe"
server_key_iv: "8b82d4848bea9b3d94a397b960b43f65"
server_key_version: "encryptionkey@0"
I20260928 15:03:58.582173 19946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:03:58.585259 19961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:58.586436 19946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:58.586599 19946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "305401fb6d4f4f1d95c3f055eb4ab549"
format_stamp: "Formatted at 2026-09-28 15:03:58 on dist-test-slave-dmhs"
server_key: "d8e30c3d7ae6236ff4692cee1e0b0dfe"
server_key_iv: "8b82d4848bea9b3d94a397b960b43f65"
server_key_version: "encryptionkey@0"
I20260928 15:03:58.586784 19946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:58.604249 19946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:58.607975 19946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:58.608204 19946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:58.609163 19946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:58.610615 19946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:58.610699 19946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:58.610790 19946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:58.610832 19946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:58.643539 19946 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:33907
I20260928 15:03:58.643580 20073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:33907 every 8 connection(s)
I20260928 15:03:58.644834 19946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:58.647431 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 19946
I20260928 15:03:58.647646 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:58.648096 9267 external_mini_cluster.cc:1428] Setting key f2c9261750cc0945de4306c4342127d4
Sep 28 15:03:58 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607838, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:03:58.664433 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.647322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:56919 (local address 127.9.12.254:37549)
0928 15:03:58.647554 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.647561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.648860 (+ 1299us) server_negotiation.cc:407] Connection header received
0928 15:03:58.651179 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.651184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.651303 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.651403 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:58.652846 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.653570 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.654994 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.655373 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.655460 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.658743 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:58.658747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:58.658750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:58.658780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:58.661131 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.661869 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.661876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.661880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.662023 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.662462 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.662467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.662478 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.662824 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:58.662981 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.663823 (+ 842us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.664059 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:03:58.665215 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.647839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:37549 (local address 127.9.12.195:56919)
0928 15:03:58.648689 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.648740 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.650883 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.651622 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.651630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.652106 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:58.652705 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.652723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.653696 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.653699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.654877 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.654889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.655565 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.655570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.655776 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.656469 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:58.656502 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:58.658570 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:58.661250 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:58.661256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:58.661268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:58.661675 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:58.662133 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:58.662139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:58.662143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:58.662340 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:58.663062 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:58.663070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:58.663387 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.663699 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.664081 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":387,"threads_started":1}
I20260928 15:03:58.666463 20074 heartbeater.cc:347] Connected to a master server at 127.9.12.254:37549
I20260928 15:03:58.666755 20074 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:58.667467 20074 heartbeater.cc:511] Master 127.9.12.254:37549 requested a full tablet report, sending...
I20260928 15:03:58.668823 19618 ts_manager.cc:194] Registered new tserver with Master: 305401fb6d4f4f1d95c3f055eb4ab549 (127.9.12.195:33907)
I20260928 15:03:58.669559 19618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:56919
I20260928 15:03:58.675617 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:58.691406 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.679721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.9.12.254:37549)
0928 15:03:58.679908 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.679916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.680440 (+ 524us) server_negotiation.cc:407] Connection header received
0928 15:03:58.680561 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.680565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.680619 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.680692 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:58.681952 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.682803 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.683902 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.684307 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.684407 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.686466 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:58.686470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:58.686472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:58.686497 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:58.688919 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.689544 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.689547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.689550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.689674 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.689986 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.689989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.689991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.690342 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:58.690522 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.690874 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.691045 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:03:58.694281 19807 heartbeater.cc:503] Master 127.9.12.254:37549 was elected leader, sending a full tablet report...
I20260928 15:03:58.695415 19618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:58.702312 19618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:58.725224 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.714410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.9.12.195:33907)
0928 15:03:58.716748 (+ 2338us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.716757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.716789 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:03:58.716913 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.716924 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.717202 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.718329 (+ 1127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:58.718603 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.719889 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.722539 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.724058 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.724189 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.724459 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.724781 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.724950 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2139,"thread_start_us":1959,"threads_started":1}
I20260928 15:03:58.725373 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.713823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:33907 (local address 127.0.0.1:36444)
0928 15:03:58.715435 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.715475 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.715637 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.717434 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.717444 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.717482 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:58.717891 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.717904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.720031 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.720039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.722315 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.722333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.724300 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.724308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.724525 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.724586 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.724729 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.724950 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"mutex_wait_us":4,"thread_start_us":2855,"threads_started":1}
I20260928 15:03:58.725977 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.713866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:32881 (local address 127.0.0.1:50464)
0928 15:03:58.716981 (+ 3115us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.717010 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.717215 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.718620 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.718627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.718662 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:58.719131 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.719146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.721063 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.721068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.723153 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.723164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.725145 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.725151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.725338 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.725384 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.725483 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.725621 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2963,"thread_start_us":1205,"threads_started":1}
I20260928 15:03:58.726735 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.714429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.9.12.193:32881)
0928 15:03:58.716161 (+ 1732us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.716169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.717107 (+ 938us) server_negotiation.cc:407] Connection header received
0928 15:03:58.717658 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.717670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.717959 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.718154 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:58.719355 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.720900 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.723355 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.724657 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.724780 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.724994 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.726210 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.726399 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1561,"thread_start_us":1404,"threads_started":1}
I20260928 15:03:58.728706 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.716499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:39945 (local address 127.0.0.1:33418)
0928 15:03:58.717267 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.717292 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.717395 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.720921 (+ 3526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.720927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.720958 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:58.721410 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.721422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.723333 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.723338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.726229 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.726243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.727912 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.727919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.728084 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.728117 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.728210 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.728354 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":899,"thread_start_us":642,"threads_started":1}
I20260928 15:03:58.728785 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.716994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.9.12.194:39945)
0928 15:03:58.720418 (+ 3424us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.720426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.720452 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:03:58.720541 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.720548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.720729 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.720918 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:58.721638 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.723159 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.726467 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.727682 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.727922 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.728111 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.728298 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.728472 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3241,"thread_start_us":3149,"threads_started":1}
I20260928 15:03:58.729537 20009 tablet_service.cc:1467] Processing CreateTablet for tablet a739509cb8ab44089960cd287f081991 (DEFAULT_TABLE table=test-table [id=bf02546c8f824a9995cc426573422043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:58.730032 19742 tablet_service.cc:1467] Processing CreateTablet for tablet a739509cb8ab44089960cd287f081991 (DEFAULT_TABLE table=test-table [id=bf02546c8f824a9995cc426573422043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:58.730755 20009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a739509cb8ab44089960cd287f081991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:58.731289 19742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a739509cb8ab44089960cd287f081991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:58.734131 19876 tablet_service.cc:1467] Processing CreateTablet for tablet a739509cb8ab44089960cd287f081991 (DEFAULT_TABLE table=test-table [id=bf02546c8f824a9995cc426573422043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:58.735437 19876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a739509cb8ab44089960cd287f081991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:58.741948 20092 tablet_bootstrap.cc:492] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Bootstrap starting.
I20260928 15:03:58.745152 20093 tablet_bootstrap.cc:492] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Bootstrap starting.
I20260928 15:03:58.745740 20092 tablet_bootstrap.cc:654] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:58.747056 20092 log.cc:824] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:58.748551 20093 tablet_bootstrap.cc:654] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:58.749306 20095 tablet_bootstrap.cc:492] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Bootstrap starting.
I20260928 15:03:58.749790 20093 log.cc:824] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:58.750612 20092 tablet_bootstrap.cc:492] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: No bootstrap required, opened a new log
I20260928 15:03:58.750874 20092 ts_tablet_manager.cc:1397] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260928 15:03:58.752655 20095 tablet_bootstrap.cc:654] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:58.752888 20093 tablet_bootstrap.cc:492] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: No bootstrap required, opened a new log
I20260928 15:03:58.753176 20093 ts_tablet_manager.cc:1397] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 15:03:58.753825 20095 log.cc:824] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:58.756070 20095 tablet_bootstrap.cc:492] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: No bootstrap required, opened a new log
I20260928 15:03:58.756366 20095 ts_tablet_manager.cc:1397] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260928 15:03:58.761842 20092 raft_consensus.cc:348] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.762327 20092 raft_consensus.cc:374] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:58.762483 20092 raft_consensus.cc:729] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2891d07f75854c8ebe79e49880dcee57, State: Initialized, Role: FOLLOWER
I20260928 15:03:58.763278 20092 consensus_queue.cc:260] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.763810 20093 raft_consensus.cc:348] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.764477 20093 raft_consensus.cc:374] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:58.764626 20093 raft_consensus.cc:729] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305401fb6d4f4f1d95c3f055eb4ab549, State: Initialized, Role: FOLLOWER
I20260928 15:03:58.764357 20095 raft_consensus.cc:348] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.764768 20095 raft_consensus.cc:374] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:58.764907 20095 raft_consensus.cc:729] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5f285e19b0450eb49073d1506ca0ed, State: Initialized, Role: FOLLOWER
I20260928 15:03:58.765275 20093 consensus_queue.cc:260] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.765475 20095 consensus_queue.cc:260] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.766407 20074 heartbeater.cc:503] Master 127.9.12.254:37549 was elected leader, sending a full tablet report...
I20260928 15:03:58.767071 20093 ts_tablet_manager.cc:1428] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 15:03:58.769225 20092 ts_tablet_manager.cc:1428] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Time spent starting tablet: real 0.018s user 0.003s sys 0.012s
I20260928 15:03:58.770198 19941 heartbeater.cc:503] Master 127.9.12.254:37549 was elected leader, sending a full tablet report...
I20260928 15:03:58.770398 20095 ts_tablet_manager.cc:1428] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260928 15:03:58.853711 20098 raft_consensus.cc:482] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:58.854003 20098 raft_consensus.cc:504] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.855841 20098 leader_election.cc:288] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305401fb6d4f4f1d95c3f055eb4ab549 (127.9.12.195:33907), bd5f285e19b0450eb49073d1506ca0ed (127.9.12.194:39945)
I20260928 15:03:58.862654 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.856402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:48453 (local address 127.9.12.195:33907)
0928 15:03:58.856655 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.856663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.857367 (+ 704us) server_negotiation.cc:407] Connection header received
0928 15:03:58.857584 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.857589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.857657 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.857757 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:58.858399 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.859146 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.860978 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.861697 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.861916 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.861991 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.862269 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.862390 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:03:58.862746 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.856402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:35377 (local address 127.9.12.194:39945)
0928 15:03:58.856655 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.856664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.857460 (+ 796us) server_negotiation.cc:407] Connection header received
0928 15:03:58.857649 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.857655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.857721 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.857818 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:58.858511 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.859208 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.861025 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.861776 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.861988 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.862067 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.862352 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.862479 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:03:58.862854 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.856137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:39945 (local address 127.9.12.193:35377)
0928 15:03:58.857346 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.857360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.857507 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.857833 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.857838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.857866 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:58.858339 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.858348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.859356 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.859361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.860873 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.860885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.862038 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.862044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.862199 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.862221 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.862325 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.862442 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":3,"thread_start_us":782,"threads_started":1}
I20260928 15:03:58.863163 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.856089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:33907 (local address 127.9.12.193:48453)
0928 15:03:58.857258 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:58.857274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:58.857437 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:58.857771 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:58.857775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:58.857799 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:58.858236 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.858245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.859271 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.859280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.860813 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:58.860827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.861932 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.861939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:58.862111 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.862134 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:03:58.862241 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:03:58.862366 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":18,"spinlock_wait_cycles":16768,"thread_start_us":967,"threads_started":1}
I20260928 15:03:58.863911 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a739509cb8ab44089960cd287f081991" candidate_uuid: "2891d07f75854c8ebe79e49880dcee57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5f285e19b0450eb49073d1506ca0ed" is_pre_election: true
I20260928 15:03:58.864048 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a739509cb8ab44089960cd287f081991" candidate_uuid: "2891d07f75854c8ebe79e49880dcee57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" is_pre_election: true
I20260928 15:03:58.864436 19896 raft_consensus.cc:2454] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2891d07f75854c8ebe79e49880dcee57 in term 0.
I20260928 15:03:58.864634 20029 raft_consensus.cc:2454] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2891d07f75854c8ebe79e49880dcee57 in term 0.
I20260928 15:03:58.865080 19698 leader_election.cc:302] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2891d07f75854c8ebe79e49880dcee57, bd5f285e19b0450eb49073d1506ca0ed; no voters:
I20260928 15:03:58.865453 20098 raft_consensus.cc:2781] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:58.865592 20098 raft_consensus.cc:482] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:58.865692 20098 raft_consensus.cc:3037] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:58.868084 20098 raft_consensus.cc:504] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.868710 20098 leader_election.cc:288] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [CANDIDATE]: Term 1 election: Requested vote from peers 305401fb6d4f4f1d95c3f055eb4ab549 (127.9.12.195:33907), bd5f285e19b0450eb49073d1506ca0ed (127.9.12.194:39945)
I20260928 15:03:58.869122 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a739509cb8ab44089960cd287f081991" candidate_uuid: "2891d07f75854c8ebe79e49880dcee57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305401fb6d4f4f1d95c3f055eb4ab549"
I20260928 15:03:58.869138 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a739509cb8ab44089960cd287f081991" candidate_uuid: "2891d07f75854c8ebe79e49880dcee57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5f285e19b0450eb49073d1506ca0ed"
I20260928 15:03:58.869309 20029 raft_consensus.cc:3037] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:58.869328 19896 raft_consensus.cc:3037] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:58.871428 20029 raft_consensus.cc:2454] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2891d07f75854c8ebe79e49880dcee57 in term 1.
I20260928 15:03:58.871507 19896 raft_consensus.cc:2454] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2891d07f75854c8ebe79e49880dcee57 in term 1.
I20260928 15:03:58.871874 19697 leader_election.cc:302] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2891d07f75854c8ebe79e49880dcee57, 305401fb6d4f4f1d95c3f055eb4ab549; no voters:
I20260928 15:03:58.872179 20098 raft_consensus.cc:2781] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:58.872612 20098 raft_consensus.cc:686] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 1 LEADER]: Becoming Leader. State: Replica: 2891d07f75854c8ebe79e49880dcee57, State: Running, Role: LEADER
I20260928 15:03:58.873131 20098 consensus_queue.cc:237] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } }
I20260928 15:03:58.877223 19618 catalog_manager.cc:5495] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2891d07f75854c8ebe79e49880dcee57 (127.9.12.193). New cstate: current_term: 1 leader_uuid: "2891d07f75854c8ebe79e49880dcee57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2891d07f75854c8ebe79e49880dcee57" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 32881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 } health_report { overall_health: UNKNOWN } } }
W20260928 15:03:58.896450 20075 tablet.cc:2369] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:58.911417 19942 tablet.cc:2369] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:58.915226 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.910069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.9.12.193:32881)
0928 15:03:58.910324 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.910329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.910377 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:03:58.910568 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.910571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.910662 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.910781 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:58.911130 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.911961 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.913270 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.913642 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.913784 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.914271 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:58.914717 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:58.914819 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.914890 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.914949 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 15:03:58.915773 19808 tablet.cc:2369] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:58.922891 20029 raft_consensus.cc:1261] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 1 FOLLOWER]: Refusing update from remote peer 2891d07f75854c8ebe79e49880dcee57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:58.922955 19896 raft_consensus.cc:1261] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 1 FOLLOWER]: Refusing update from remote peer 2891d07f75854c8ebe79e49880dcee57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:58.923971 20103 consensus_queue.cc:1035] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5f285e19b0450eb49073d1506ca0ed" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:58.924463 20098 consensus_queue.cc:1035] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "305401fb6d4f4f1d95c3f055eb4ab549" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:58.936883 20106 mvcc.cc:205] Tried to move back new op lower bound from 7334329708218970112 to 7334329708027924480. Current Snapshot: MvccSnapshot[applied={T|T < 7334329708218970112}]
I20260928 15:03:58.940250 20107 mvcc.cc:205] Tried to move back new op lower bound from 7334329708218970112 to 7334329708027924480. Current Snapshot: MvccSnapshot[applied={T|T < 7334329708218970112}]
I20260928 15:03:58.941507 20108 mvcc.cc:205] Tried to move back new op lower bound from 7334329708218970112 to 7334329708027924480. Current Snapshot: MvccSnapshot[applied={T|T < 7334329708218970112 or (T in {7334329708218970112})}]
I20260928 15:03:58.953186 19617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:58.953454 19617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:58.957015 19617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd21fab45b894ca7aca070ea227660a2: Sending DeleteTablet for 3 replicas of tablet a739509cb8ab44089960cd287f081991
I20260928 15:03:58.958158 20009 tablet_service.cc:1514] Processing DeleteTablet for tablet a739509cb8ab44089960cd287f081991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:58 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:36444
I20260928 15:03:58.958333 19742 tablet_service.cc:1514] Processing DeleteTablet for tablet a739509cb8ab44089960cd287f081991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:58 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:50464
I20260928 15:03:58.958655 19876 tablet_service.cc:1514] Processing DeleteTablet for tablet a739509cb8ab44089960cd287f081991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:58 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:33418
I20260928 15:03:58.961036 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19679
I20260928 15:03:58.970657 20115 tablet_replica.cc:330] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: stopping tablet replica
I20260928 15:03:58.971354 20117 tablet_replica.cc:330] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: stopping tablet replica
I20260928 15:03:58.972038 20117 raft_consensus.cc:2229] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:58.972679 20115 raft_consensus.cc:2229] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:58.973398 20117 raft_consensus.cc:2258] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:58.973588 20116 tablet_replica.cc:330] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: stopping tablet replica
I20260928 15:03:58.973912 20115 raft_consensus.cc:2258] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:58.974073 20116 raft_consensus.cc:2229] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:58.974838 20116 raft_consensus.cc:2258] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:58.976948 20117 ts_tablet_manager.cc:1905] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:58.976958 20116 ts_tablet_manager.cc:1905] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:58.982306 20115 ts_tablet_manager.cc:1905] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:58.984730 20117 ts_tablet_manager.cc:1918] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:58.984787 20116 ts_tablet_manager.cc:1918] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:58.984938 20117 log.cc:1196] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/a739509cb8ab44089960cd287f081991
I20260928 15:03:58.985026 20116 log.cc:1196] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/a739509cb8ab44089960cd287f081991
I20260928 15:03:58.985591 20117 ts_tablet_manager.cc:1939] T a739509cb8ab44089960cd287f081991 P bd5f285e19b0450eb49073d1506ca0ed: Deleting consensus metadata
I20260928 15:03:58.985697 20116 ts_tablet_manager.cc:1939] T a739509cb8ab44089960cd287f081991 P 2891d07f75854c8ebe79e49880dcee57: Deleting consensus metadata
I20260928 15:03:58.987510 19605 catalog_manager.cc:4841] TS bd5f285e19b0450eb49073d1506ca0ed (127.9.12.194:39945): tablet a739509cb8ab44089960cd287f081991 (table test-table [id=bf02546c8f824a9995cc426573422043]) successfully deleted
I20260928 15:03:58.987838 19605 catalog_manager.cc:4841] TS 2891d07f75854c8ebe79e49880dcee57 (127.9.12.193:32881): tablet a739509cb8ab44089960cd287f081991 (table test-table [id=bf02546c8f824a9995cc426573422043]) successfully deleted
I20260928 15:03:58.990216 20115 ts_tablet_manager.cc:1918] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 15:03:58 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:03:58.990391 20115 log.cc:1196] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/a739509cb8ab44089960cd287f081991
I20260928 15:03:58.990914 20115 ts_tablet_manager.cc:1939] T a739509cb8ab44089960cd287f081991 P 305401fb6d4f4f1d95c3f055eb4ab549: Deleting consensus metadata
I20260928 15:03:58.992759 19604 catalog_manager.cc:4841] TS 305401fb6d4f4f1d95c3f055eb4ab549 (127.9.12.195:33907): tablet a739509cb8ab44089960cd287f081991 (table test-table [id=bf02546c8f824a9995cc426573422043]) successfully deleted
I20260928 15:03:58.997771 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:58.962801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.9.12.193:32881)
0928 15:03:58.963044 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:58.963053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:58.973876 (+ 10823us) server_negotiation.cc:407] Connection header received
0928 15:03:58.974099 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:58.974105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:58.974195 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:58.974309 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:58.976086 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.977233 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.979029 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:58.979830 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:58.979937 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:58.991560 (+ 11623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:58.991575 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:58.991592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:58.991639 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:58.995029 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.995857 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.995863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.995865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.996015 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:58.996457 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:58.996460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:58.996462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:58.996822 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:58.997037 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:58.997348 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 15:03:58.997517 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:03:58.998483 19802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:59.095211 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19679
I20260928 15:03:59.113510 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19813
Sep 28 15:03:59 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:03:59.128003 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:59.114350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.9.12.194:39945)
0928 15:03:59.114518 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:59.114522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:59.114540 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:59.114658 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:59.114662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:59.114722 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:59.114813 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:59.116078 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:59.116976 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:59.118156 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:59.118539 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:59.118637 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:59.122859 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:59.122876 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:59.122899 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:59.122931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:59.125650 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:59.126248 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:59.126255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:59.126257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:59.126374 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:59.126679 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:59.126682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:59.126683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:59.127164 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:59.127432 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:59.127594 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 15:03:59.127751 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:03:59.128823 19936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:59.216607 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19813
I20260928 15:03:59.234616 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19946
Sep 28 15:03:59 dist-test-slave-dmhs krb5kdc[19573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:03:59.249173 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:59.235398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.9.12.195:33907)
0928 15:03:59.235584 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:59.235589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:59.235606 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:03:59.235721 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:59.235725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:59.235786 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:59.235866 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:59.237219 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:59.237924 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:59.239095 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:59.239471 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:59.239560 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:59.243839 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:59.243850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:59.243864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:59.243894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:59.246490 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:59.247077 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:59.247084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:59.247086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:59.247215 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:59.247540 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:59.247543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:59.247545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:59.247869 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:59.248093 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:59.248724 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 15:03:59.248915 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:03:59.249851 20069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:59.339659 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19946
I20260928 15:03:59.355984 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 19586
I20260928 15:03:59.357146 19648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:03:59.470064 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 19586
2026-09-28T15:03:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4123 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:59 dist-test-slave-dmhs krb5kdc[20125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:59 dist-test-slave-dmhs krb5kdc[20125](info): set up 2 sockets
Sep 28 15:03:59 dist-test-slave-dmhs krb5kdc[20125](info): commencing operation
krb5kdc: starting...
W20260928 15:04:00.565704 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:04:00 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:04:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/rangeradmin_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/rangeradmin_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/rangerlookup_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/rangerlookup_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
I20260928 15:04:00.877044 9267 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7VZsZQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/postgres -l logfile start
2026-09-28 15:04:03.312 UTC [20155] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:04:03.312 UTC [20155] LOG: listening on IPv4 address "127.9.12.212", port 48499
2026-09-28 15:04:03.315 UTC [20155] LOG: listening on Unix socket "/tmp/.s.PGSQL.48499"
2026-09-28 15:04:03.336 UTC [20158] LOG: database system was shut down at 2026-09-28 15:04:01 UTC
2026-09-28 15:04:03.339 UTC [20155] LOG: database system is ready to accept connections
I20260928 15:04:04.289119 9267 mini_postgres.cc:96] Postgres bound to 48499
2026-09-28 15:04:04.307 UTC [20166] FATAL: database "slave" does not exist
127.9.12.212:48499 - accepting connections
I20260928 15:04:04.307999 9267 mini_ranger.cc:161] Starting Ranger...
I20260928 15:04:04.333292 9267 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:04:04.780467 9267 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:04:04.780606 9267 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-admin
2026-09-28 15:04:05,070 [I] DB FLAVOR :POSTGRES
2026-09-28 15:04:05,070 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:04:05,071 [I] Checking connection..
2026-09-28 15:04:05,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:04:05,332 [I] Checking connection passed.
2026-09-28 15:04:05,333 [I] --------- Verifying version history table ---------
2026-09-28 15:04:05,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:05,563 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:04:05,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:05,786 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:04:05,786 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:04:05,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:04:06.028 UTC [20230] WARNING: there is no transaction in progress
2026-09-28 15:04:06,034 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:04:06,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:06,288 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:04:06,288 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:04:06,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:04:06,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:04:06,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dmhs.c.gcp-upstream.internal','N') ;"
2026-09-28 15:04:07,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:04:07,269 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:04:07,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:04:07,523 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:04:07,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:04:07,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:04:08,011 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:04:08,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:04:08.905 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:08.920 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:08.936 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:08.950 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:08.965 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:08.984 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.046 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.052 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.065 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.075 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.085 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.098 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.107 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.112 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.118 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09.124 UTC [20338] WARNING: there is no transaction in progress
2026-09-28 15:04:09,694 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:04:09,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:04:09,942 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:04:09,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:04:10,176 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:04:10,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:04:10,409 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:04:10,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:04:10,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:04:10,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dmhs.c.gcp-upstream.internal';"
2026-09-28 15:04:11,146 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:04:11,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:04:11,370 [I] DB_PATCHES have already been applied
I20260928 15:04:11.376178 9267 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-admin:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7VZsZQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:04:11.376307 9267 mini_ranger.cc:193] Using host: 127.9.12.212
I20260928 15:04:11.391945 9267 mini_ranger.cc:227] Ranger admin URL: http://127.9.12.212:45035
Sep 28, 2026 3:04:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:04:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:04:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:04:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:04:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:04:12.528119 9267 mini_ranger.cc:160] Time spent starting Ranger: real 8.220s user 0.010s sys 0.029s
Sep 28, 2026 3:04:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:04:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 3:04:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:04:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:04:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:04:33.142639 9267 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/rangerkms_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/rangerkms_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.12.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.12.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:04:33 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607873, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:04:33.320888 9267 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:04:33.342154 9267 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:04:34.202819 9267 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:04:34.203012 9267 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms
I20260928 15:04:34.203168 9267 mini_ranger_kms.cc:227] Using postgres at 127.9.12.212:48499
2026-09-28 15:04:34,897 [I] DB FLAVOR :POSTGRES
2026-09-28 15:04:34,897 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:04:34,897 [I] Checking connection
2026-09-28 15:04:34,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:04:35,130 [I] connection success
2026-09-28 15:04:35,130 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:04:35,130 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:04:35,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:04:35,351 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:04:35,352 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:04:35,352 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:04:35,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:48499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:04:35,598 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:04:36.076648 9267 mini_ranger_kms.cc:310] Created kms service
I20260928 15:04:36.238072 9267 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:04:36.281430 9267 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:04:36 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607873, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:04:36.697068 9267 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:04:36.697420 9267 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms:/tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:04:36.697491 9267 mini_ranger_kms.cc:240] Using host: 127.9.12.212
I20260928 15:04:36.720458 9267 mini_ranger_kms.cc:276] Ranger KMS PID: 20518
I20260928 15:04:36.720675 9267 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.12.212:42245
Sep 28, 2026 3:04:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:04:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:04:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:04:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:04:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:04:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:04:38.976963 9267 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.656s user 0.117s sys 0.360s
I20260928 15:04:38.977226 9267 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:04:38.977434 9267 mini_ranger_kms.cc:401] 127.9.12.212:42245/kms/v1/keys
Sep 28 15:04:39 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607879, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.12.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:04:39 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607879, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.12.212@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
Sep 28 15:04:39 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607879, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.12.212@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
Sep 28 15:04:42 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:04:42.170164 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:38973
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:38973
--ranger_config_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42245/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:04:42.546216 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:42.546556 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:42.546752 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:42.554876 20569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:04:42.555161 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:42.555325 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:42.555495 20569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:04:42.555642 20569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:04:42.567782 20569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.12.212:42245/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:38973
--ranger_config_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:04:42.569214 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:42.570717 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:42.587275 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:42.587845 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:42.588151 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:42.588666 20569 server_base.cc:1034] running on GCE node
I20260928 15:04:42.589473 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:42.590895 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:42.592129 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790607882592099 us; error 43 us; skew 500 ppm
I20260928 15:04:42.592514 20569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:04:42 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:42.595530 20569 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:04:42.597100 20569 webserver.cc:460] Webserver started at http://127.9.12.254:46321/ using document root <none> and password file <none>
I20260928 15:04:42.597783 20569 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:42.597927 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:42.598248 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:42 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:04:42.664605 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8"
format_stamp: "Formatted at 2026-09-28 15:04:42 on dist-test-slave-dmhs"
server_key: "04513d6f24631111d2a40226bbb94d6d"
server_key_iv: "23fa0e46b1b17aac6d25d877d178bf4c"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:42.665717 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8"
format_stamp: "Formatted at 2026-09-28 15:04:42 on dist-test-slave-dmhs"
server_key: "04513d6f24631111d2a40226bbb94d6d"
server_key_iv: "23fa0e46b1b17aac6d25d877d178bf4c"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:42.672854 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260928 15:04:42.693940 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:42.695700 20569 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:04:42.695971 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8"
format_stamp: "Formatted at 2026-09-28 15:04:42 on dist-test-slave-dmhs"
server_key: "04513d6f24631111d2a40226bbb94d6d"
server_key_iv: "23fa0e46b1b17aac6d25d877d178bf4c"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:42.696317 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:42.712225 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:42.716543 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:42.716851 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:42.743522 20569 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:38973
I20260928 15:04:42.743556 20635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:38973 every 8 connection(s)
I20260928 15:04:42.744990 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:04:42.749210 20636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:42.752048 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 20569
I20260928 15:04:42.752295 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:04:42.753114 9267 external_mini_cluster.cc:1428] Setting key 2e7b17450e493b3bf88e280c91936747
I20260928 15:04:42.765125 20636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8: Bootstrap starting.
I20260928 15:04:42.770355 20636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:42.771595 20636 log.cc:824] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:42.775420 20636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8: No bootstrap required, opened a new log
Sep 28 15:04:42 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:04:42.786883 20636 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38973 } }
I20260928 15:04:42.787307 20636 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:42.787475 20636 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f1facb306d4c3bbb3e5cd03d550ec8, State: Initialized, Role: FOLLOWER
I20260928 15:04:42.788177 20636 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38973 } }
I20260928 15:04:42.788515 20636 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:04:42.788699 20636 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:04:42.788911 20636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:42.791554 20636 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38973 } }
I20260928 15:04:42.792163 20636 leader_election.cc:302] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f1facb306d4c3bbb3e5cd03d550ec8; no voters:
I20260928 15:04:42.792927 20636 leader_election.cc:288] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:04:42.793057 20641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:04:42.799242 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:42.755387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.9.12.254:38973)
0928 15:04:42.770151 (+ 14764us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:42.770169 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:42.770230 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:04:42.772939 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:42.772974 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:42.773497 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:42.773983 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:42.775813 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:42.778390 (+ 2577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:42.780366 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:42.781510 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:42.781706 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:42.787513 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:42.787527 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:42.787549 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:42.787603 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:42.791808 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:42.792821 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:42.792834 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:42.792843 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:42.793062 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:42.793575 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:42.793584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:42.793590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:42.794522 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:42.794868 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:42.796712 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0928 15:04:42.797336 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14479,"thread_start_us":14654,"threads_started":1}
I20260928 15:04:42.800597 20641 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [term 1 LEADER]: Becoming Leader. State: Replica: e3f1facb306d4c3bbb3e5cd03d550ec8, State: Running, Role: LEADER
I20260928 15:04:42.801748 20636 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:04:42.802917 20641 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38973 } }
I20260928 15:04:42.820393 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38973 } } }
I20260928 15:04:42.821025 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:04:42.821136 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f1facb306d4c3bbb3e5cd03d550ec8. Latest consensus state: current_term: 1 leader_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1facb306d4c3bbb3e5cd03d550ec8" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 38973 } } }
I20260928 15:04:42.821475 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:04:42.822468 20657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:04:42.822623 20636 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:04:42.829667 20657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:04:42.862287 20657 catalog_manager.cc:1347] Generated new cluster ID: 0425bfc73fa94599ab16c0fd6e59808a
I20260928 15:04:42.862416 20657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:04:42.886771 20657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:04:42.888861 20657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:04:42.903501 20657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8: Generated new TSK 0
I20260928 15:04:42.904716 20657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:04:43.672446 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:38973
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:04:43 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607883, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 15:04:44.088330 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:44.088665 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:44.088846 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:44.096818 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:44.096979 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:44.097155 20682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:04:44.108986 20682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.12.212:42245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:04:44.110579 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:44.112067 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:44.131122 20693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:44.131844 20692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:44.138257 20695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:44.139258 20682 server_base.cc:1034] running on GCE node
I20260928 15:04:44.140134 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:44.141932 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:44.143327 20682 hybrid_clock.cc:648] HybridClock initialized: now 1790607884143160 us; error 95 us; skew 500 ppm
I20260928 15:04:44.143877 20682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:04:44 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607884, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:44.148800 20682 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:04:44.157795 20682 webserver.cc:460] Webserver started at http://127.9.12.193:42087/ using document root <none> and password file <none>
I20260928 15:04:44.158806 20682 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:44.159019 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:44.159397 20682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:44 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607884, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:04:44.185453 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "d4ff24593ae04755ad19a755d12ac727"
format_stamp: "Formatted at 2026-09-28 15:04:44 on dist-test-slave-dmhs"
server_key: "007b062a53a65e9115c1982549c86d5a"
server_key_iv: "562405f2e5ba0b1d56ffc8d2baf89379"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:44.186506 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "d4ff24593ae04755ad19a755d12ac727"
format_stamp: "Formatted at 2026-09-28 15:04:44 on dist-test-slave-dmhs"
server_key: "007b062a53a65e9115c1982549c86d5a"
server_key_iv: "562405f2e5ba0b1d56ffc8d2baf89379"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:44.194706 20682 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260928 15:04:44.224648 20702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:44.226969 20682 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.007s
I20260928 15:04:44.227283 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "d4ff24593ae04755ad19a755d12ac727"
format_stamp: "Formatted at 2026-09-28 15:04:44 on dist-test-slave-dmhs"
server_key: "007b062a53a65e9115c1982549c86d5a"
server_key_iv: "562405f2e5ba0b1d56ffc8d2baf89379"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:44.227696 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:44.286849 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:44.291292 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:44.291741 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:44.292881 20682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:04:44.294394 20682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:04:44.294500 20682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:44.294649 20682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:04:44.294725 20682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:44.386025 20682 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:45429
I20260928 15:04:44.387843 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:04:44.385993 20814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:45429 every 8 connection(s)
I20260928 15:04:44.397051 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 20682
I20260928 15:04:44.397264 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:04:44.398087 9267 external_mini_cluster.cc:1428] Setting key 2a512c00798c74bb3febb20f63e24770
Sep 28 15:04:44 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607884, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:04:44.452443 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:44.394093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:38973 (local address 127.9.12.193:58633)
0928 15:04:44.425107 (+ 31014us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:44.425212 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:44.430266 (+ 5054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:44.431715 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:44.431729 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:44.432475 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:04:44.433461 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:44.433493 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:44.435416 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:44.435422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:44.438140 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:44.438156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:44.439569 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:44.439576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:44.439855 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:44.440697 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:04:44.440727 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:04:44.443587 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:04:44.447741 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:44.447750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:44.447769 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:44.448370 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:44.448771 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:44.448775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:44.448779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:44.449004 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:44.449665 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:04:44.449672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:04:44.450080 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 15:04:44.450348 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 15:04:44.450875 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30529,"thread_start_us":34081,"threads_started":1}
I20260928 15:04:44.453754 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:44.393572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:58633 (local address 127.9.12.254:38973)
0928 15:04:44.431113 (+ 37541us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:44.431118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:44.431135 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:04:44.431187 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:44.431192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:44.431264 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:44.431399 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:44.433729 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:44.435186 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:44.438411 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:44.439126 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:44.439251 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:44.443879 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:44.443887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:44.443892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:44.443943 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:44.447546 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:44.448518 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:44.448524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:44.448527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:44.448656 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:44.449124 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:44.449129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:44.449133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:44.449471 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:44.449598 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:44.453140 (+ 3542us) server_negotiation.cc:299] Negotiation successful
0928 15:04:44.453382 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37421,"thread_start_us":37295,"threads_started":1}
I20260928 15:04:44.455974 20815 heartbeater.cc:347] Connected to a master server at 127.9.12.254:38973
I20260928 15:04:44.456386 20815 heartbeater.cc:464] Registering TS with master...
I20260928 15:04:44.457235 20815 heartbeater.cc:511] Master 127.9.12.254:38973 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:04:44.459776 20601 ts_manager.cc:194] Registered new tserver with Master: d4ff24593ae04755ad19a755d12ac727 (127.9.12.193:45429)
I20260928 15:04:44.461880 20601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:58633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:04:44.495920 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:38973
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:04:44 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607883, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
Sep 28 15:04:44 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607883, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
W20260928 15:04:44.996655 20821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:44.997215 20821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:44.997414 20821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:45.009593 20821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:45.009838 20821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:45.010190 20821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:04:45.024413 20821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.12.212:42245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:04:45.026010 20821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:45.027490 20821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:45.048520 20829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:45.048775 20828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:45.050295 20831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:45.055287 20821 server_base.cc:1034] running on GCE node
I20260928 15:04:45.056079 20821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:45.064319 20821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:45.065650 20821 hybrid_clock.cc:648] HybridClock initialized: now 1790607885065539 us; error 100 us; skew 500 ppm
I20260928 15:04:45.066040 20821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:04:45 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:45.069646 20821 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:04:45.074729 20821 webserver.cc:460] Webserver started at http://127.9.12.194:45001/ using document root <none> and password file <none>
I20260928 15:04:45.075479 20821 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:45.075639 20821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:45.075976 20821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:45 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:04:45.093695 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "ff5b6139e5bf49c58f94ab42178bec1e"
format_stamp: "Formatted at 2026-09-28 15:04:45 on dist-test-slave-dmhs"
server_key: "92ec5c3d89f66ddb258b422d461a2b9d"
server_key_iv: "8e040caab9cf5d637e791930e540acce"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:45.094656 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "ff5b6139e5bf49c58f94ab42178bec1e"
format_stamp: "Formatted at 2026-09-28 15:04:45 on dist-test-slave-dmhs"
server_key: "92ec5c3d89f66ddb258b422d461a2b9d"
server_key_iv: "8e040caab9cf5d637e791930e540acce"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:45.101631 20821 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 15:04:45.122617 20845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:45.124766 20821 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:04:45.124980 20821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "ff5b6139e5bf49c58f94ab42178bec1e"
format_stamp: "Formatted at 2026-09-28 15:04:45 on dist-test-slave-dmhs"
server_key: "92ec5c3d89f66ddb258b422d461a2b9d"
server_key_iv: "8e040caab9cf5d637e791930e540acce"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:45.125269 20821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:45.154814 20821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:45.160166 20821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:45.160970 20821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:45.162248 20821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:04:45.164055 20821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:04:45.164180 20821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:45.164347 20821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:04:45.164438 20821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:45.217502 20821 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:39675
I20260928 15:04:45.217535 20957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:39675 every 8 connection(s)
I20260928 15:04:45.218868 20821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:04:45.226423 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 20821
I20260928 15:04:45.226599 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:04:45.227149 9267 external_mini_cluster.cc:1428] Setting key b8c67617a3dc47f10fa168076c3001b7
Sep 28 15:04:45 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:04:45.241370 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.221654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:51953 (local address 127.9.12.254:38973)
0928 15:04:45.222137 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:45.222141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:45.223201 (+ 1060us) server_negotiation.cc:407] Connection header received
0928 15:04:45.225281 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:45.225287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:45.225396 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:45.225519 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:45.227095 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.228046 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.230151 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.230699 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.230824 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.234615 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:45.234626 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:45.234629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:45.234658 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:45.237754 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:45.238637 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:45.238643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:45.238647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:45.238797 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:45.239340 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:45.239346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:45.239349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:45.239679 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:45.239838 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:45.240683 (+ 845us) server_negotiation.cc:299] Negotiation successful
0928 15:04:45.240895 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":287,"threads_started":1}
I20260928 15:04:45.242494 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.222257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:38973 (local address 127.9.12.194:51953)
0928 15:04:45.223042 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:45.223078 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:45.225012 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:45.225756 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:45.225764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:45.226192 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:04:45.226900 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.226938 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.228217 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.228227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.230010 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.230022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.231097 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.231102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.231297 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.232134 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:04:45.232156 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:04:45.234404 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:04:45.237906 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:45.237914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:45.237930 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:45.238487 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:45.238925 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:45.238933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:45.238939 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:45.239190 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:45.239812 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:04:45.239822 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:04:45.240221 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 15:04:45.240557 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 15:04:45.241121 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":339,"threads_started":1}
I20260928 15:04:45.244021 20958 heartbeater.cc:347] Connected to a master server at 127.9.12.254:38973
I20260928 15:04:45.244396 20958 heartbeater.cc:464] Registering TS with master...
I20260928 15:04:45.245182 20958 heartbeater.cc:511] Master 127.9.12.254:38973 requested a full tablet report, sending...
I20260928 15:04:45.246744 20601 ts_manager.cc:194] Registered new tserver with Master: ff5b6139e5bf49c58f94ab42178bec1e (127.9.12.194:39675)
I20260928 15:04:45.247984 20601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:51953
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:04:45.288381 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:38973
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:04:45.465957 20815 heartbeater.cc:503] Master 127.9.12.254:38973 was elected leader, sending a full tablet report...
W20260928 15:04:45.639875 20964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:45.640126 20964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:45.640249 20964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:45.647989 20964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:45.648152 20964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:45.648438 20964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:04:45.658962 20964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.12.212:42245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:04:45.660301 20964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:45.661527 20964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:45.676102 20970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:45.676362 20969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:45.676337 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:45.677410 20964 server_base.cc:1034] running on GCE node
I20260928 15:04:45.677980 20964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:45.678936 20964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:45.680136 20964 hybrid_clock.cc:648] HybridClock initialized: now 1790607885680070 us; error 58 us; skew 500 ppm
I20260928 15:04:45.680536 20964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:04:45 dist-test-slave-dmhs krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:45.683861 20964 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:04:45.685448 20964 webserver.cc:460] Webserver started at http://127.9.12.195:39795/ using document root <none> and password file <none>
I20260928 15:04:45.686146 20964 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:45.686245 20964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:45.686517 20964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:45 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:04:45.703104 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "3a2a5715eab0444cbbe8c728119d8a70"
format_stamp: "Formatted at 2026-09-28 15:04:45 on dist-test-slave-dmhs"
server_key: "261d674c9354ba43d949d7e602c5417e"
server_key_iv: "8f28d843b4e9aad015a1fda9ba8aa227"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:45.704210 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "3a2a5715eab0444cbbe8c728119d8a70"
format_stamp: "Formatted at 2026-09-28 15:04:45 on dist-test-slave-dmhs"
server_key: "261d674c9354ba43d949d7e602c5417e"
server_key_iv: "8f28d843b4e9aad015a1fda9ba8aa227"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:45.710238 20964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:04:45.725581 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:45.727113 20964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:04:45.727365 20964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "3a2a5715eab0444cbbe8c728119d8a70"
format_stamp: "Formatted at 2026-09-28 15:04:45 on dist-test-slave-dmhs"
server_key: "261d674c9354ba43d949d7e602c5417e"
server_key_iv: "8f28d843b4e9aad015a1fda9ba8aa227"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:45.727733 20964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:45.742756 20964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:45.746797 20964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:45.747160 20964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:45.748072 20964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:04:45.749883 20964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:04:45.750092 20964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:45.750268 20964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:04:45.750346 20964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:45.787547 20964 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:43001
I20260928 15:04:45.787590 21091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:43001 every 8 connection(s)
I20260928 15:04:45.789006 20964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:04:45.792634 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 20964
I20260928 15:04:45.792850 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:04:45.793507 9267 external_mini_cluster.cc:1428] Setting key 0c374d66b97e9069f363fdcc28ef6b54
Sep 28 15:04:45 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:04:45.811280 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.791652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:43343 (local address 127.9.12.254:38973)
0928 15:04:45.792237 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:45.792244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:45.793587 (+ 1343us) server_negotiation.cc:407] Connection header received
0928 15:04:45.797006 (+ 3419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:45.797012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:45.797141 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:45.797258 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:45.798694 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.799433 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.801142 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.801806 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.802055 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.805457 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:45.805466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:45.805475 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:45.805507 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:45.807942 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:45.808868 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:45.808874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:45.808878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:45.809026 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:45.809599 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:45.809603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:45.809605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:45.809865 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:45.810030 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:45.810679 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 15:04:45.810903 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":235,"threads_started":1}
I20260928 15:04:45.812197 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.792299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:38973 (local address 127.9.12.195:43343)
0928 15:04:45.793385 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:45.793444 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:45.796704 (+ 3260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:45.797503 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:45.797511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:45.797938 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:04:45.798507 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.798525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.799595 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.799600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.800993 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.801005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.802143 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.802150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.802404 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.803065 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:04:45.803101 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:04:45.805234 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:04:45.808087 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:45.808096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:45.808112 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:45.808741 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:45.809141 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:45.809146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:45.809150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:45.809370 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:45.810003 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:04:45.810011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:04:45.810322 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 15:04:45.810585 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 15:04:45.811101 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":402,"threads_started":1}
I20260928 15:04:45.813736 21092 heartbeater.cc:347] Connected to a master server at 127.9.12.254:38973
I20260928 15:04:45.814121 21092 heartbeater.cc:464] Registering TS with master...
I20260928 15:04:45.815147 21092 heartbeater.cc:511] Master 127.9.12.254:38973 requested a full tablet report, sending...
I20260928 15:04:45.816747 20601 ts_manager.cc:194] Registered new tserver with Master: 3a2a5715eab0444cbbe8c728119d8a70 (127.9.12.195:43001)
I20260928 15:04:45.817898 20601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:43343
I20260928 15:04:45.821792 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:04:45.840241 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.826659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.9.12.254:38973)
0928 15:04:45.826923 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:45.826932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:45.827650 (+ 718us) server_negotiation.cc:407] Connection header received
0928 15:04:45.827807 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:45.827810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:45.827907 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:45.827985 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:45.829666 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.830542 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.832004 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.832564 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.832651 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.835164 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:45.835170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:45.835173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:45.835219 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:45.837876 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:45.838567 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:45.838570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:45.838573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:45.838687 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:45.839014 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:45.839017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:45.839019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:45.839264 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:45.839441 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:45.839701 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 15:04:45.839931 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:04:45.844480 20601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:04:45.852176 20601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:04:45.889966 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.866007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.9.12.193:45429)
0928 15:04:45.873058 (+ 7051us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:45.873072 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:45.878057 (+ 4985us) server_negotiation.cc:407] Connection header received
0928 15:04:45.878388 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:45.878400 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:45.878760 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:45.879014 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:45.881523 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.883259 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.886986 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.888109 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.888265 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.888620 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:45.889187 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 15:04:45.889429 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":428,"threads_started":1}
I20260928 15:04:45.890010 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.866082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45429 (local address 127.0.0.1:46408)
0928 15:04:45.877886 (+ 11804us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:45.877946 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:45.878174 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:45.880066 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:45.880077 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:45.880144 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:45.881366 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.881379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.883464 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.883469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.886756 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.886767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.888433 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.888440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.888877 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.888944 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 15:04:45.889104 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 15:04:45.889295 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11485,"mutex_wait_us":61,"thread_start_us":11301,"threads_started":1}
I20260928 15:04:45.893554 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.866773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43001 (local address 127.0.0.1:44044)
0928 15:04:45.880641 (+ 13868us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:45.880677 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:45.880861 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:45.882863 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:45.882868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:45.882894 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:45.883394 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.883409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.885855 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.885866 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.889188 (+ 3322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.889202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.892255 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.892286 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.892543 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.892567 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:04:45.892695 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 15:04:45.892820 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13731,"mutex_wait_us":41,"thread_start_us":14912,"threads_started":1}
I20260928 15:04:45.893549 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.867043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.9.12.195:43001)
0928 15:04:45.882344 (+ 15301us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:45.882349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:45.882374 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:04:45.882447 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:45.882451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:45.882601 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:45.882887 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:45.883688 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.885698 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.890338 (+ 4640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.891973 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.892143 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.892465 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:45.892758 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 15:04:45.892911 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15159,"thread_start_us":14993,"threads_started":1}
I20260928 15:04:45.895153 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.866946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:39675 (local address 127.0.0.1:48662)
0928 15:04:45.883715 (+ 16769us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:45.883745 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:45.883977 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:45.885083 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:45.885088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:45.885130 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:45.886036 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.886049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.888868 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.888873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.891913 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:45.891929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.894624 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.894629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:45.894739 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.894761 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:04:45.894834 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 15:04:45.894939 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14937,"thread_start_us":13154,"threads_started":1}
I20260928 15:04:45.895591 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:45.867182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.9.12.194:39675)
0928 15:04:45.881184 (+ 14002us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:45.881192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:45.884395 (+ 3203us) server_negotiation.cc:407] Connection header received
0928 15:04:45.884531 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:45.884546 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:45.884848 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:45.885039 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:45.886376 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.888687 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.892085 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:45.894391 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:45.894572 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:45.894770 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:45.894870 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 15:04:45.895060 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13846,"thread_start_us":14033,"threads_started":1}
I20260928 15:04:45.895248 20750 tablet_service.cc:1467] Processing CreateTablet for tablet 31ed61bae05945c09ad44c557c55ca55 (DEFAULT_TABLE table=test-table [id=a2fa58b8cf854abc9f43e9707a6533f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:04:45.897289 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ed61bae05945c09ad44c557c55ca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:45.898756 21027 tablet_service.cc:1467] Processing CreateTablet for tablet 31ed61bae05945c09ad44c557c55ca55 (DEFAULT_TABLE table=test-table [id=a2fa58b8cf854abc9f43e9707a6533f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:04:45.899909 21027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ed61bae05945c09ad44c557c55ca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:45.905809 20893 tablet_service.cc:1467] Processing CreateTablet for tablet 31ed61bae05945c09ad44c557c55ca55 (DEFAULT_TABLE table=test-table [id=a2fa58b8cf854abc9f43e9707a6533f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:04:45.907259 20893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ed61bae05945c09ad44c557c55ca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:45.912742 21111 tablet_bootstrap.cc:492] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Bootstrap starting.
I20260928 15:04:45.917037 21111 tablet_bootstrap.cc:654] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:45.919178 21111 log.cc:824] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:45.920475 21112 tablet_bootstrap.cc:492] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Bootstrap starting.
I20260928 15:04:45.923774 21111 tablet_bootstrap.cc:492] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: No bootstrap required, opened a new log
I20260928 15:04:45.924256 21112 tablet_bootstrap.cc:654] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:45.924412 21111 ts_tablet_manager.cc:1397] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260928 15:04:45.925079 21114 tablet_bootstrap.cc:492] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Bootstrap starting.
I20260928 15:04:45.925645 21112 log.cc:824] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:45.928512 21112 tablet_bootstrap.cc:492] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: No bootstrap required, opened a new log
I20260928 15:04:45.928769 21112 ts_tablet_manager.cc:1397] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 15:04:45.928867 21114 tablet_bootstrap.cc:654] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:45.930092 21114 log.cc:824] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:45.937309 21112 raft_consensus.cc:348] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:45.937286 21111 raft_consensus.cc:348] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:45.937709 21112 raft_consensus.cc:374] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:45.937795 21112 raft_consensus.cc:729] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ff24593ae04755ad19a755d12ac727, State: Initialized, Role: FOLLOWER
I20260928 15:04:45.937830 21111 raft_consensus.cc:374] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:45.937958 21111 raft_consensus.cc:729] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2a5715eab0444cbbe8c728119d8a70, State: Initialized, Role: FOLLOWER
I20260928 15:04:45.938534 21112 consensus_queue.cc:260] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:45.938675 21111 consensus_queue.cc:260] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:45.941167 21112 ts_tablet_manager.cc:1428] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260928 15:04:45.941915 21114 tablet_bootstrap.cc:492] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: No bootstrap required, opened a new log
I20260928 15:04:45.942196 21114 ts_tablet_manager.cc:1397] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20260928 15:04:45.945902 21092 heartbeater.cc:503] Master 127.9.12.254:38973 was elected leader, sending a full tablet report...
I20260928 15:04:45.946830 21111 ts_tablet_manager.cc:1428] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Time spent starting tablet: real 0.022s user 0.016s sys 0.005s
I20260928 15:04:45.951102 21114 raft_consensus.cc:348] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:45.951448 21114 raft_consensus.cc:374] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:45.951565 21114 raft_consensus.cc:729] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5b6139e5bf49c58f94ab42178bec1e, State: Initialized, Role: FOLLOWER
I20260928 15:04:45.952162 21114 consensus_queue.cc:260] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:45.953212 20958 heartbeater.cc:503] Master 127.9.12.254:38973 was elected leader, sending a full tablet report...
I20260928 15:04:45.953661 21114 ts_tablet_manager.cc:1428] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
W20260928 15:04:45.971004 20959 tablet.cc:2369] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:04:46.040655 21093 tablet.cc:2369] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:04:46.073858 21119 raft_consensus.cc:482] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:04:46.074199 21119 raft_consensus.cc:504] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:46.075845 21119 leader_election.cc:288] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4ff24593ae04755ad19a755d12ac727 (127.9.12.193:45429), 3a2a5715eab0444cbbe8c728119d8a70 (127.9.12.195:43001)
I20260928 15:04:46.084837 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.076670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:60947 (local address 127.9.12.193:45429)
0928 15:04:46.076933 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:46.076939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:46.078047 (+ 1108us) server_negotiation.cc:407] Connection header received
0928 15:04:46.078379 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:46.078387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:46.078516 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:46.078636 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:46.079338 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.080640 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.082811 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.083942 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.084067 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.084130 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:46.084566 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 15:04:46.084655 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:04:46.085145 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.076574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:45429 (local address 127.9.12.194:60947)
0928 15:04:46.077901 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:46.077927 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:46.078123 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:46.078680 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:46.078688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:46.078738 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:46.079156 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:46.079167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.080774 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.080788 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:46.082650 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:46.082661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.084192 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.084198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:46.084396 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.084429 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:04:46.084555 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:04:46.084693 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":1,"thread_start_us":1855,"threads_started":1}
I20260928 15:04:46.086292 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.076532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:41865 (local address 127.9.12.195:43001)
0928 15:04:46.076851 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:46.076857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:46.079033 (+ 2176us) server_negotiation.cc:407] Connection header received
0928 15:04:46.079163 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:46.079168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:46.079261 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:46.079343 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:46.080367 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.081671 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.083704 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.085266 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.085451 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.085525 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:46.085818 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 15:04:46.085941 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:04:46.086352 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.076671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43001 (local address 127.9.12.194:41865)
0928 15:04:46.078896 (+ 2225us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:46.078926 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:46.079052 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:46.079380 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:46.079385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:46.079428 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:46.080179 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:46.080192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.081837 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.081840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:46.083547 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:46.083561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.085484 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.085489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:46.085657 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.085679 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:04:46.085808 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 15:04:46.085928 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"spinlock_wait_cycles":23424,"thread_start_us":770,"threads_started":1}
I20260928 15:04:46.086380 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ed61bae05945c09ad44c557c55ca55" candidate_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ff24593ae04755ad19a755d12ac727" is_pre_election: true
I20260928 15:04:46.087472 20770 raft_consensus.cc:2454] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5b6139e5bf49c58f94ab42178bec1e in term 0.
I20260928 15:04:46.087600 21047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ed61bae05945c09ad44c557c55ca55" candidate_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2a5715eab0444cbbe8c728119d8a70" is_pre_election: true
I20260928 15:04:46.088207 21047 raft_consensus.cc:2454] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5b6139e5bf49c58f94ab42178bec1e in term 0.
I20260928 15:04:46.088486 20848 leader_election.cc:302] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4ff24593ae04755ad19a755d12ac727, ff5b6139e5bf49c58f94ab42178bec1e; no voters:
I20260928 15:04:46.089027 21119 raft_consensus.cc:2781] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:04:46.089244 21119 raft_consensus.cc:482] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:04:46.089387 21119 raft_consensus.cc:3037] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:46.091756 21119 raft_consensus.cc:504] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:46.092540 21119 leader_election.cc:288] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [CANDIDATE]: Term 1 election: Requested vote from peers d4ff24593ae04755ad19a755d12ac727 (127.9.12.193:45429), 3a2a5715eab0444cbbe8c728119d8a70 (127.9.12.195:43001)
I20260928 15:04:46.093075 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ed61bae05945c09ad44c557c55ca55" candidate_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ff24593ae04755ad19a755d12ac727"
I20260928 15:04:46.093119 21047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ed61bae05945c09ad44c557c55ca55" candidate_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2a5715eab0444cbbe8c728119d8a70"
I20260928 15:04:46.093331 21047 raft_consensus.cc:3037] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:46.093371 20770 raft_consensus.cc:3037] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:46.096437 21047 raft_consensus.cc:2454] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5b6139e5bf49c58f94ab42178bec1e in term 1.
I20260928 15:04:46.096469 20770 raft_consensus.cc:2454] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5b6139e5bf49c58f94ab42178bec1e in term 1.
I20260928 15:04:46.096936 20847 leader_election.cc:302] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2a5715eab0444cbbe8c728119d8a70, ff5b6139e5bf49c58f94ab42178bec1e; no voters:
I20260928 15:04:46.097329 21119 raft_consensus.cc:2781] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:04:46.097896 21119 raft_consensus.cc:686] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 1 LEADER]: Becoming Leader. State: Replica: ff5b6139e5bf49c58f94ab42178bec1e, State: Running, Role: LEADER
I20260928 15:04:46.098471 21119 consensus_queue.cc:237] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } }
I20260928 15:04:46.103646 20600 catalog_manager.cc:5495] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5b6139e5bf49c58f94ab42178bec1e (127.9.12.194). New cstate: current_term: 1 leader_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5b6139e5bf49c58f94ab42178bec1e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 39675 } health_report { overall_health: HEALTHY } } }
W20260928 15:04:46.143558 20816 tablet.cc:2369] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:04:46.175994 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.169746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.9.12.194:39675)
0928 15:04:46.169951 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:46.169956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:46.169974 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:04:46.170117 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:46.170120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:46.170191 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:46.170335 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:04:46.170666 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.171598 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.173436 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.174002 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.174153 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.174769 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:04:46.175259 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:04:46.175460 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:46.175511 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 15:04:46.175602 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:04:46.184962 20770 raft_consensus.cc:1261] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 1 FOLLOWER]: Refusing update from remote peer ff5b6139e5bf49c58f94ab42178bec1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:04:46.185019 21047 raft_consensus.cc:1261] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 1 FOLLOWER]: Refusing update from remote peer ff5b6139e5bf49c58f94ab42178bec1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:04:46.185946 21122 consensus_queue.cc:1035] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4ff24593ae04755ad19a755d12ac727" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:04:46.186262 21119 consensus_queue.cc:1035] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2a5715eab0444cbbe8c728119d8a70" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:04:46.202904 21125 mvcc.cc:205] Tried to move back new op lower bound from 7334329901802627072 to 7334329901463756800. Current Snapshot: MvccSnapshot[applied={T|T < 7334329901802627072 or (T in {7334329901802627072})}]
I20260928 15:04:46.207571 21126 mvcc.cc:205] Tried to move back new op lower bound from 7334329901802627072 to 7334329901463756800. Current Snapshot: MvccSnapshot[applied={T|T < 7334329901802627072}]
I20260928 15:04:46.217833 20600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:04:46.218133 20600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:04:46.222582 20600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f1facb306d4c3bbb3e5cd03d550ec8: Sending DeleteTablet for 3 replicas of tablet 31ed61bae05945c09ad44c557c55ca55
I20260928 15:04:46.223934 20750 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ed61bae05945c09ad44c557c55ca55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:04:46 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:46408
I20260928 15:04:46.224136 20893 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ed61bae05945c09ad44c557c55ca55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:04:46 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48662
I20260928 15:04:46.224911 21027 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ed61bae05945c09ad44c557c55ca55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:04:46 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:44044
I20260928 15:04:46.225224 21138 tablet_replica.cc:330] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: stopping tablet replica
I20260928 15:04:46.225747 21138 raft_consensus.cc:2229] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:04:46.226517 21138 raft_consensus.cc:2258] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:04:46.228914 21138 ts_tablet_manager.cc:1905] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:04:46.232489 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 20682
I20260928 15:04:46.237995 21138 ts_tablet_manager.cc:1918] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:04:46.238173 21138 log.cc:1196] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/31ed61bae05945c09ad44c557c55ca55
I20260928 15:04:46.238796 21138 ts_tablet_manager.cc:1939] T 31ed61bae05945c09ad44c557c55ca55 P ff5b6139e5bf49c58f94ab42178bec1e: Deleting consensus metadata
I20260928 15:04:46.240950 20585 catalog_manager.cc:4841] TS ff5b6139e5bf49c58f94ab42178bec1e (127.9.12.194:39675): tablet 31ed61bae05945c09ad44c557c55ca55 (table test-table [id=a2fa58b8cf854abc9f43e9707a6533f6]) successfully deleted
I20260928 15:04:46.244809 21139 tablet_replica.cc:330] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: stopping tablet replica
I20260928 15:04:46.245718 21139 raft_consensus.cc:2229] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:04:46.246439 21137 tablet_replica.cc:330] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: stopping tablet replica
I20260928 15:04:46.246799 21137 raft_consensus.cc:2229] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:04:46.247289 21137 raft_consensus.cc:2258] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:04:46.248039 21139 raft_consensus.cc:2258] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:04:46.249976 21137 ts_tablet_manager.cc:1905] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:04:46.250528 21139 ts_tablet_manager.cc:1905] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:04:46.257783 21139 ts_tablet_manager.cc:1918] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:04:46.257782 21137 ts_tablet_manager.cc:1918] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:04:46.257956 21139 log.cc:1196] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/31ed61bae05945c09ad44c557c55ca55
I20260928 15:04:46.258132 21137 log.cc:1196] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/31ed61bae05945c09ad44c557c55ca55
I20260928 15:04:46.258564 21139 ts_tablet_manager.cc:1939] T 31ed61bae05945c09ad44c557c55ca55 P 3a2a5715eab0444cbbe8c728119d8a70: Deleting consensus metadata
I20260928 15:04:46.258805 21137 ts_tablet_manager.cc:1939] T 31ed61bae05945c09ad44c557c55ca55 P d4ff24593ae04755ad19a755d12ac727: Deleting consensus metadata
I20260928 15:04:46.260483 20586 catalog_manager.cc:4841] TS 3a2a5715eab0444cbbe8c728119d8a70 (127.9.12.195:43001): tablet 31ed61bae05945c09ad44c557c55ca55 (table test-table [id=a2fa58b8cf854abc9f43e9707a6533f6]) successfully deleted
I20260928 15:04:46.260655 20587 catalog_manager.cc:4841] TS d4ff24593ae04755ad19a755d12ac727 (127.9.12.193:45429): tablet 31ed61bae05945c09ad44c557c55ca55 (table test-table [id=a2fa58b8cf854abc9f43e9707a6533f6]) successfully deleted
Sep 28 15:04:46 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:04:46.274878 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.234097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.9.12.193:45429)
0928 15:04:46.234266 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:46.234272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:46.248882 (+ 14610us) server_negotiation.cc:407] Connection header received
0928 15:04:46.252726 (+ 3844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:46.252733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:46.252831 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:46.252913 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:46.255016 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.256253 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.258389 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.259108 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.259205 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.268037 (+ 8832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:46.268068 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:46.268087 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:46.268138 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:46.271897 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:46.272702 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:46.272711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:46.272715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:46.272896 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:46.273253 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:46.273258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:46.273261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:46.273758 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:46.274061 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:46.274229 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 15:04:46.274463 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:04:46.275617 20810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:04:46.367058 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 20682
I20260928 15:04:46.384217 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 20821
Sep 28 15:04:46 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:04:46.402248 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.385186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.9.12.194:39675)
0928 15:04:46.385410 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:46.385415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:46.385435 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:04:46.385501 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:46.385506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:46.385577 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:46.385687 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:46.386853 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.388101 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.389556 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.390197 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.390318 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.394893 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:46.394914 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:46.394935 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:46.394990 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:46.399133 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:46.399904 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:46.399914 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:46.399918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:46.400138 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:46.400573 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:46.400578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:46.400581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:46.401034 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:46.401369 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:46.401621 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:04:46.401877 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:04:46.403132 20953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:04:46.520080 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 20821
I20260928 15:04:46.534265 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 20964
Sep 28 15:04:46 dist-test-slave-dmhs krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:04:46.548645 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:46.535054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.9.12.195:43001)
0928 15:04:46.535227 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:46.535233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:46.535332 (+ 99us) server_negotiation.cc:407] Connection header received
0928 15:04:46.535541 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:46.535546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:46.535622 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:46.535735 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:46.537076 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.537780 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.539077 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:46.539436 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:46.539588 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:46.543746 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:46.543759 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:46.543775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:46.543808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:46.546343 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:46.546909 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:46.546915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:46.546917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:46.547060 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:46.547357 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:46.547360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:46.547361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:46.547707 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:46.547923 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:46.548178 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 15:04:46.548395 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:04:46.549273 21087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:04:46.636646 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 20964
I20260928 15:04:46.652659 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 20569
I20260928 15:04:46.653846 20631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:04:46.766518 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 20569
I20260928 15:04:46.788214 9267 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:04:47.122834 9267 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:04:47.123090 9267 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:04:47.469017 9267 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:04:47.469 UTC [20155] LOG: received smart shutdown request
2026-09-28 15:04:47.473 UTC [20155] LOG: background worker "logical replication launcher" (PID 20164) exited with exit code 1
2026-09-28 15:04:47.473 UTC [20159] LOG: shutting down
2026-09-28 15:04:47.631 UTC [20155] LOG: database system is shut down
2026-09-28T15:04:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48290 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:04:47 dist-test-slave-dmhs krb5kdc[21153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:04:47 dist-test-slave-dmhs krb5kdc[21153](info): set up 2 sockets
Sep 28 15:04:47 dist-test-slave-dmhs krb5kdc[21153](info): commencing operation
krb5kdc: starting...
W20260928 15:04:48.852922 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:04:48 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:04:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/rangeradmin_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/rangeradmin_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/rangerlookup_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/rangerlookup_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
I20260928 15:04:49.170409 9267 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7VZsZQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/postgres -l logfile start
2026-09-28 15:04:51.588 UTC [21183] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:04:51.588 UTC [21183] LOG: listening on IPv4 address "127.9.12.212", port 54423
2026-09-28 15:04:51.591 UTC [21183] LOG: listening on Unix socket "/tmp/.s.PGSQL.54423"
2026-09-28 15:04:51.612 UTC [21186] LOG: database system was shut down at 2026-09-28 15:04:50 UTC
2026-09-28 15:04:51.616 UTC [21183] LOG: database system is ready to accept connections
I20260928 15:04:52.676678 9267 mini_postgres.cc:96] Postgres bound to 54423
2026-09-28 15:04:52.700 UTC [21194] FATAL: database "slave" does not exist
127.9.12.212:54423 - accepting connections
I20260928 15:04:52.701179 9267 mini_ranger.cc:161] Starting Ranger...
I20260928 15:04:52.727381 9267 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:04:53.177299 9267 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:04:53.177418 9267 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-admin
2026-09-28 15:04:53,462 [I] DB FLAVOR :POSTGRES
2026-09-28 15:04:53,463 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:04:53,463 [I] Checking connection..
2026-09-28 15:04:53,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:04:53,690 [I] Checking connection passed.
2026-09-28 15:04:53,690 [I] --------- Verifying version history table ---------
2026-09-28 15:04:53,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:53,937 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:04:53,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:54,177 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:04:54,177 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:04:54,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:04:54.407 UTC [21258] WARNING: there is no transaction in progress
2026-09-28 15:04:54,413 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:04:54,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:54,677 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:04:54,677 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:04:54,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:04:54,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:04:55,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dmhs.c.gcp-upstream.internal','N') ;"
2026-09-28 15:04:55,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:04:55,609 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:04:55,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:04:55,848 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:04:55,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:04:56,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:04:56,333 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:04:56,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:04:57.259 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.274 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.290 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.304 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.320 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.336 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.389 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.396 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.411 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.420 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.430 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.443 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.452 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.458 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.465 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:57.471 UTC [21366] WARNING: there is no transaction in progress
2026-09-28 15:04:58,090 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:04:58,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:04:58,312 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:04:58,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:04:58,558 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:04:58,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:04:58,789 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:04:58,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:04:59,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:04:59,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dmhs.c.gcp-upstream.internal';"
2026-09-28 15:04:59,515 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:04:59,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:04:59,759 [I] DB_PATCHES have already been applied
I20260928 15:04:59.764905 9267 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-admin:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7VZsZQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:04:59.764992 9267 mini_ranger.cc:193] Using host: 127.9.12.212
I20260928 15:04:59.780093 9267 mini_ranger.cc:227] Ranger admin URL: http://127.9.12.212:60189
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:05:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:05:00.907312 9267 mini_ranger.cc:160] Time spent starting Ranger: real 8.206s user 0.004s sys 0.033s
Sep 28, 2026 3:05:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:05:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 3:05:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:05:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:05:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:05:22.475145 9267 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.12.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.12.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/rangerkms_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.12.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/rangerkms_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.12.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.12.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.12.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/HTTP_127.9.12.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:05:22 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607922, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:05:22.642146 9267 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:05:22.664485 9267 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:05:23.488209 9267 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:05:23.488358 9267 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms
I20260928 15:05:23.488407 9267 mini_ranger_kms.cc:227] Using postgres at 127.9.12.212:54423
2026-09-28 15:05:23,846 [I] DB FLAVOR :POSTGRES
2026-09-28 15:05:23,846 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:05:23,846 [I] Checking connection
2026-09-28 15:05:23,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:05:24,076 [I] connection success
2026-09-28 15:05:24,076 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:05:24,077 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:05:24,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:05:24,303 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:05:24,303 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:05:24,303 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:05:24,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.12.212:54423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:05:24,555 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:05:25.023725 9267 mini_ranger_kms.cc:310] Created kms service
I20260928 15:05:25.235101 9267 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:05:25.280180 9267 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:05:25 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607922, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:25.710822 9267 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:05:25.711149 9267 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms:/tmp/dist-test-task7VZsZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7VZsZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:05:25.711223 9267 mini_ranger_kms.cc:240] Using host: 127.9.12.212
I20260928 15:05:25.727597 9267 mini_ranger_kms.cc:276] Ranger KMS PID: 21546
I20260928 15:05:25.727738 9267 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.12.212:42215
Sep 28, 2026 3:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:05:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:05:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:05:28.019263 9267 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.377s user 0.071s sys 0.302s
I20260928 15:05:28.019526 9267 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:05:28.019711 9267 mini_ranger_kms.cc:401] 127.9.12.212:42215/kms/v1/keys
Sep 28 15:05:28 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607928, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.12.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:05:28 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607928, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.12.212@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
Sep 28 15:05:28 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607928, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.12.212@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
Sep 28 15:05:31 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:05:31.253643 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43687
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:43687
--ranger_config_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:31.593442 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:31.593799 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:31.594009 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:31.601336 21597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:31.601503 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:31.601613 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:31.601718 21597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:31.601814 21597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:31.612042 21597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.12.212:42215/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:43687
--ranger_config_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:31.613435 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:31.614784 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:31.626740 21602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:31.626735 21603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:31.626814 21605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:31.627604 21597 server_base.cc:1034] running on GCE node
I20260928 15:05:31.628432 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:31.629884 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:31.631134 21597 hybrid_clock.cc:648] HybridClock initialized: now 1790607931631099 us; error 45 us; skew 500 ppm
I20260928 15:05:31.631492 21597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:31 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:31.634618 21597 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:31.636045 21597 webserver.cc:460] Webserver started at http://127.9.12.254:36737/ using document root <none> and password file <none>
I20260928 15:05:31.636736 21597 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:31.636830 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:31.637090 21597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:31 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:31.693934 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "a963e91154a642f1aa3f85bc2b1ea588"
format_stamp: "Formatted at 2026-09-28 15:05:31 on dist-test-slave-dmhs"
server_key: "a864b1bc9fff4a957f04b36ff573540a"
server_key_iv: "af2f2c7635385c35cfb927ea957109b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:31.694800 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "a963e91154a642f1aa3f85bc2b1ea588"
format_stamp: "Formatted at 2026-09-28 15:05:31 on dist-test-slave-dmhs"
server_key: "a864b1bc9fff4a957f04b36ff573540a"
server_key_iv: "af2f2c7635385c35cfb927ea957109b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:31.700335 21597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:05:31.719108 21612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:31.721043 21597 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260928 15:05:31.721254 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "a963e91154a642f1aa3f85bc2b1ea588"
format_stamp: "Formatted at 2026-09-28 15:05:31 on dist-test-slave-dmhs"
server_key: "a864b1bc9fff4a957f04b36ff573540a"
server_key_iv: "af2f2c7635385c35cfb927ea957109b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:31.721509 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:31.739665 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:31.743538 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:31.743868 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:31.763084 21597 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:43687
I20260928 15:05:31.763104 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:43687 every 8 connection(s)
I20260928 15:05:31.764631 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:31.768117 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 21597
I20260928 15:05:31.768368 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:05:31.769191 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:31.769275 9267 external_mini_cluster.cc:1428] Setting key 824e9b96b5d560bf552e9945df597e20
I20260928 15:05:31.784416 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Bootstrap starting.
Sep 28 15:05:31 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:31.787662 21664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:31.788851 21664 log.cc:824] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:31.791633 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: No bootstrap required, opened a new log
I20260928 15:05:31.792940 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:31.771415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.9.12.254:43687)
0928 15:05:31.772336 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:31.772348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:31.772398 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:05:31.774252 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:31.774276 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:31.774699 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:31.775063 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:31.776775 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:31.778069 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:31.779962 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:31.780768 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:31.780906 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:31.785602 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:31.785610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:31.785625 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:31.785668 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:31.788842 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:31.789579 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:31.789586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:31.789596 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:31.789764 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:31.790159 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:31.790165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:31.790170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:31.790696 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:31.791019 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:31.791453 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 15:05:31.791873 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":398,"threads_started":1}
I20260928 15:05:31.799207 21664 raft_consensus.cc:348] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:31.799575 21664 raft_consensus.cc:374] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:31.799683 21664 raft_consensus.cc:729] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a963e91154a642f1aa3f85bc2b1ea588, State: Initialized, Role: FOLLOWER
I20260928 15:05:31.800242 21664 consensus_queue.cc:260] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:31.800521 21664 raft_consensus.cc:388] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:31.800673 21664 raft_consensus.cc:482] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:31.800836 21664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:31.803330 21664 raft_consensus.cc:504] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:31.803802 21664 leader_election.cc:302] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a963e91154a642f1aa3f85bc2b1ea588; no voters:
I20260928 15:05:31.804414 21664 leader_election.cc:288] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:05:31.804505 21669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:31.805258 21669 raft_consensus.cc:686] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 LEADER]: Becoming Leader. State: Replica: a963e91154a642f1aa3f85bc2b1ea588, State: Running, Role: LEADER
I20260928 15:05:31.805861 21669 consensus_queue.cc:237] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:31.806638 21664 sys_catalog.cc:564] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:31.808928 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a963e91154a642f1aa3f85bc2b1ea588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } } }
I20260928 15:05:31.809297 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:31.809051 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a963e91154a642f1aa3f85bc2b1ea588. Latest consensus state: current_term: 1 leader_uuid: "a963e91154a642f1aa3f85bc2b1ea588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } } }
I20260928 15:05:31.809435 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:31.811753 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:31.817402 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:31.818204 21664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:05:31.838709 21675 catalog_manager.cc:1347] Generated new cluster ID: 63b5adb21e804973b02ae30e0bb03a31
I20260928 15:05:31.838963 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:31.858968 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:31.861377 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:31.876770 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Generated new TSK 0
I20260928 15:05:31.878082 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:05:32.592032 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43687
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:32 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607932, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 15:05:33.021091 21709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:33.021495 21709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:33.021745 21709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:33.033627 21709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:33.033778 21709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:33.034044 21709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:05:33.049860 21709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.12.212:42215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:33.051715 21709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:33.053488 21709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:33.079208 21720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:33.079641 21719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:33.084434 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:33.084430 21709 server_base.cc:1034] running on GCE node
I20260928 15:05:33.085479 21709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:33.091024 21709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:33.092341 21709 hybrid_clock.cc:648] HybridClock initialized: now 1790607933092221 us; error 66 us; skew 500 ppm
I20260928 15:05:33.092724 21709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:33 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607933, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:33.096170 21709 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:05:33.097949 21709 webserver.cc:460] Webserver started at http://127.9.12.193:39801/ using document root <none> and password file <none>
I20260928 15:05:33.098704 21709 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:33.098832 21709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:33.099169 21709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:33 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607933, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:33.132978 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "b8914bb88e7841959a1f5acbe96146a1"
format_stamp: "Formatted at 2026-09-28 15:05:33 on dist-test-slave-dmhs"
server_key: "4550bb78abd4b41d1350ee4a806e0a65"
server_key_iv: "69ead1ff9808fe0275855a262a193a10"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:33.134430 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "b8914bb88e7841959a1f5acbe96146a1"
format_stamp: "Formatted at 2026-09-28 15:05:33 on dist-test-slave-dmhs"
server_key: "4550bb78abd4b41d1350ee4a806e0a65"
server_key_iv: "69ead1ff9808fe0275855a262a193a10"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:33.141808 21709 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 15:05:33.169675 21729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:33.171653 21709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:05:33.171901 21709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "b8914bb88e7841959a1f5acbe96146a1"
format_stamp: "Formatted at 2026-09-28 15:05:33 on dist-test-slave-dmhs"
server_key: "4550bb78abd4b41d1350ee4a806e0a65"
server_key_iv: "69ead1ff9808fe0275855a262a193a10"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:33.172246 21709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:33.219918 21709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:33.225502 21709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:33.225931 21709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:33.233371 21709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:33.241647 21709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:33.241817 21709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:05:33.242033 21709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:33.242166 21709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:05:33 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607932, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:33.742678 21709 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:36981
I20260928 15:05:33.742756 21841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:36981 every 8 connection(s)
I20260928 15:05:33.744280 21709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:33.752252 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 21709
I20260928 15:05:33.752506 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:05:33.753355 9267 external_mini_cluster.cc:1428] Setting key 6f7a915281fe9e37397ac460aa44204f
Sep 28 15:05:33 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607933, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
I20260928 15:05:33.796269 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:33.747219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:46493 (local address 127.9.12.254:43687)
0928 15:05:33.775857 (+ 28638us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:33.775863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:33.775895 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:05:33.775964 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:33.775969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:33.776065 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:33.776211 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:33.778825 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:33.779997 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:33.782524 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:33.783111 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:33.783232 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:33.788094 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:33.788101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:33.788106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:33.788151 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:33.791784 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:33.792823 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:33.792829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:33.792833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:33.792979 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:33.793528 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:33.793534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:33.793537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:33.793885 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:33.794015 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:33.795606 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0928 15:05:33.795834 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28491,"thread_start_us":28321,"threads_started":1}
I20260928 15:05:33.797127 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:33.747788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43687 (local address 127.9.12.193:46493)
0928 15:05:33.759171 (+ 11383us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:33.759243 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:33.763057 (+ 3814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:33.776832 (+ 13775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:33.776847 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:33.777597 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:33.778569 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:33.778602 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:33.780202 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:33.780208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:33.782315 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:33.782330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:33.783575 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:33.783583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:33.783886 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:33.784789 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:33.784824 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:33.787775 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:33.791982 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:33.791992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:33.792011 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:33.792661 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:33.793100 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:33.793104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:33.793108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:33.793375 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:33.794107 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:33.794116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:33.794675 (+ 559us) client_negotiation.cc:769] Sending connection context
0928 15:05:33.795037 (+ 362us) client_negotiation.cc:240] Negotiation successful
0928 15:05:33.795594 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11011,"thread_start_us":13773,"threads_started":1}
I20260928 15:05:33.799350 21842 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43687
I20260928 15:05:33.799695 21842 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:33.800580 21842 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:33.806082 21629 ts_manager.cc:194] Registered new tserver with Master: b8914bb88e7841959a1f5acbe96146a1 (127.9.12.193:36981)
I20260928 15:05:33.808053 21629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:46493
Sep 28 15:05:33 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607932, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:05:33.840812 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43687
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:34.295022 21849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:34.295444 21849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:34.295606 21849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:34.307216 21849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:34.307368 21849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:34.307566 21849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:05:34.323177 21849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.12.212:42215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:34.325006 21849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:34.326789 21849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:34.340462 21863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:34.341204 21865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:34.340453 21862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:34.345698 21849 server_base.cc:1034] running on GCE node
I20260928 15:05:34.346345 21849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:34.347422 21849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:34.348634 21849 hybrid_clock.cc:648] HybridClock initialized: now 1790607934348574 us; error 54 us; skew 500 ppm
I20260928 15:05:34.348984 21849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:34 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607934, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:34.352595 21849 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:05:34.354075 21849 webserver.cc:460] Webserver started at http://127.9.12.194:39491/ using document root <none> and password file <none>
I20260928 15:05:34.354818 21849 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:34.354965 21849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:34.355299 21849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:34 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607934, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:34.372454 21849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "dda682f8139b40318213750a57ff20a8"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "93fc7e285406f4fe3d6f268e5055f4f0"
server_key_iv: "6659dff5fe6ad27d010029a1b9153097"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:34.373582 21849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "dda682f8139b40318213750a57ff20a8"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "93fc7e285406f4fe3d6f268e5055f4f0"
server_key_iv: "6659dff5fe6ad27d010029a1b9153097"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:34.380790 21849 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 15:05:34.397116 21872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:34.398663 21849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:34.398833 21849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "dda682f8139b40318213750a57ff20a8"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "93fc7e285406f4fe3d6f268e5055f4f0"
server_key_iv: "6659dff5fe6ad27d010029a1b9153097"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:34.399057 21849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:34.424971 21849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:34.429296 21849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:34.429636 21849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:34.430590 21849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:34.431953 21849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:34.432026 21849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:34.432173 21849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:34.432233 21849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:34.469378 21849 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:37527
I20260928 15:05:34.469386 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:37527 every 8 connection(s)
I20260928 15:05:34.470707 21849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:34.479151 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 21849
I20260928 15:05:34.479338 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:05:34.479806 9267 external_mini_cluster.cc:1428] Setting key b9d654027e2cded417450ca47a7fdeda
Sep 28 15:05:34 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607934, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:34.491775 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:34.473342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:51053 (local address 127.9.12.254:43687)
0928 15:05:34.473782 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:34.473786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:34.474846 (+ 1060us) server_negotiation.cc:407] Connection header received
0928 15:05:34.476645 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:34.476650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:34.476718 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:34.476814 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:34.478254 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:34.479105 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:34.481046 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:34.481495 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:34.481613 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:34.485482 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:34.485490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:34.485494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:34.485527 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:34.488256 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:34.489262 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:34.489266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:34.489269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:34.489388 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:34.489925 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:34.489930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:34.489934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:34.490236 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:34.490398 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:34.491138 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 15:05:34.491383 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260928 15:05:34.492815 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:34.473893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43687 (local address 127.9.12.194:51053)
0928 15:05:34.474697 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:34.474734 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:34.476388 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:34.477012 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:34.477019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:34.477449 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:34.478095 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:34.478119 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:34.479266 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:34.479288 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:34.480884 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:34.480900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:34.481726 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:34.481734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:34.482019 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:34.482734 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:34.482756 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:34.485261 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:34.488453 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:34.488465 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:34.488486 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:34.489143 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:34.489492 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:34.489500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:34.489506 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:34.489770 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:34.490373 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:34.490382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:34.490746 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 15:05:34.491017 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:05:34.491574 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":280,"threads_started":1}
I20260928 15:05:34.494273 21985 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43687
I20260928 15:05:34.494608 21985 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:34.495456 21985 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:34.497109 21629 ts_manager.cc:194] Registered new tserver with Master: dda682f8139b40318213750a57ff20a8 (127.9.12.194:37527)
I20260928 15:05:34.497901 21629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:51053
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:05:34.535362 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43687
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:34.812103 21842 heartbeater.cc:503] Master 127.9.12.254:43687 was elected leader, sending a full tablet report...
W20260928 15:05:34.866282 21991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:34.866622 21991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:34.866809 21991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:34.874558 21991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:34.874718 21991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:34.874893 21991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:05:34.885170 21991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.12.212:42215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:34.886608 21991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:34.887944 21991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:34.904902 21996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:34.904837 21997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:34.905018 21999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:34.905876 21991 server_base.cc:1034] running on GCE node
I20260928 15:05:34.906539 21991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:34.907641 21991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:34.908888 21991 hybrid_clock.cc:648] HybridClock initialized: now 1790607934908818 us; error 63 us; skew 500 ppm
I20260928 15:05:34.909222 21991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:34 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607934, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:34.912635 21991 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:05:34.914242 21991 webserver.cc:460] Webserver started at http://127.9.12.195:39995/ using document root <none> and password file <none>
I20260928 15:05:34.914920 21991 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:34.915021 21991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:34.915280 21991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:34 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607934, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:34.931519 21991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "a875d781ef1c4cf5a6b41a117afb253a"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "cee869d0cff4d029f9ef94a69fceee15"
server_key_iv: "0b6bb3ec105422130670d21987162296"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:34.932512 21991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "a875d781ef1c4cf5a6b41a117afb253a"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "cee869d0cff4d029f9ef94a69fceee15"
server_key_iv: "0b6bb3ec105422130670d21987162296"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:34.937973 21991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:05:34.952299 22006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:34.953929 21991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:05:34.954110 21991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "a875d781ef1c4cf5a6b41a117afb253a"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "cee869d0cff4d029f9ef94a69fceee15"
server_key_iv: "0b6bb3ec105422130670d21987162296"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:34.954327 21991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:34.973461 21991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:34.977380 21991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:34.977691 21991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:34.978754 21991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:34.980608 21991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:34.980764 21991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:34.980969 21991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:34.981091 21991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:35.028280 21991 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:46367
I20260928 15:05:35.028340 22118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:46367 every 8 connection(s)
I20260928 15:05:35.029688 21991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:35.039080 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 21991
I20260928 15:05:35.039301 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:05:35.039794 9267 external_mini_cluster.cc:1428] Setting key e4c243fae5defa03d3c5be8cb5e4c43f
Sep 28 15:05:35 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607934, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:35.050154 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:35.032534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:39549 (local address 127.9.12.254:43687)
0928 15:05:35.033106 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:35.033111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:35.033848 (+ 737us) server_negotiation.cc:407] Connection header received
0928 15:05:35.035894 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:35.035900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:35.035990 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:35.036114 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:35.037747 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.038519 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.040441 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.040967 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.041143 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:35.044915 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:35.044920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:35.044924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:35.044956 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:35.047441 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.048114 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.048118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.048120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.048238 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.048674 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.048678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.048679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.048952 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:35.049062 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:35.049675 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 15:05:35.049835 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":313,"threads_started":1}
I20260928 15:05:35.051142 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:35.032959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43687 (local address 127.9.12.195:39549)
0928 15:05:35.033692 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:35.033730 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:35.035594 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:35.036380 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:35.036388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:35.036831 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:35.037414 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:35.037431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.038701 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.038709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:35.040294 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:35.040312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.041243 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.041250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:35.041572 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:35.042364 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:35.042390 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:35.044717 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:35.047567 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:35.047574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:35.047587 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:35.048009 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:35.048394 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:35.048397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:35.048399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:35.048576 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:35.049052 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:35.049057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:35.049374 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 15:05:35.049595 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 15:05:35.049978 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":318,"threads_started":1}
I20260928 15:05:35.052640 22119 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43687
I20260928 15:05:35.052910 22119 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:35.053709 22119 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:35.055254 21629 ts_manager.cc:194] Registered new tserver with Master: a875d781ef1c4cf5a6b41a117afb253a (127.9.12.195:46367)
I20260928 15:05:35.055958 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:35.056300 21629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:39549
I20260928 15:05:35.058018 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 21709
Sep 28 15:05:35 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:05:35.074633 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:35.058970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.9.12.193:36981)
0928 15:05:35.059737 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:35.059743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:35.059767 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:05:35.059835 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:35.059842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:35.060025 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:35.060153 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:35.061594 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.062606 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.063966 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.064560 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.064652 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:35.069002 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:35.069015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:35.069029 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:35.069064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:35.072029 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.072744 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.072751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.072753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.072882 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.073243 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.073246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.073248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.073576 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:35.073807 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:35.074108 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:05:35.074311 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":434,"threads_started":1}
I20260928 15:05:35.076035 21837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:35.160871 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 21709
I20260928 15:05:35.176816 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 21849
Sep 28 15:05:35 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:05:35.192703 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:35.177843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.9.12.194:37527)
0928 15:05:35.178495 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:35.178502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:35.178527 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:05:35.178604 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:35.178610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:35.178811 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:35.178949 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:35.180175 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.181271 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.182677 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.183231 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.183336 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:35.187546 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:35.187557 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:35.187571 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:35.187606 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:35.190239 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.190863 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.190870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.190872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.191006 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.191329 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.191332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.191334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.191659 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:35.191889 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:35.192149 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 15:05:35.192409 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":352,"threads_started":1}
I20260928 15:05:35.194044 21980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:35.277647 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 21849
I20260928 15:05:35.293633 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 21991
Sep 28 15:05:35 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:05:35.310585 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:35.294792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.9.12.195:46367)
0928 15:05:35.295453 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:35.295460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:35.295489 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:05:35.295571 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:35.295577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:35.295791 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:35.295928 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:35.297299 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.298287 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.299538 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:35.300074 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:35.300168 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:35.304755 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:35.304766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:35.304780 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:35.304813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:35.307554 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.308365 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.308377 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.308383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.308550 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:35.308940 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:35.308945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:35.308949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:35.309302 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:35.309663 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:35.309961 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 15:05:35.310224 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":360,"threads_started":1}
I20260928 15:05:35.311856 22114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:35.394635 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 21991
I20260928 15:05:35.410624 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 21597
I20260928 15:05:35.411943 21659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:35.520730 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 21597
I20260928 15:05:35.541962 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43687
--webserver_interface=127.9.12.254
--webserver_port=36737
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:43687
--ranger_config_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:35.866235 22131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:35.866528 22131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:35.866677 22131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:35.874090 22131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:35.874199 22131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:35.874300 22131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:35.874399 22131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:35.874486 22131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:35.884559 22131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:43687
--ranger_config_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=36737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:35.885890 22131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:35.887272 22131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:35.899379 22136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:35.899282 22139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:35.899385 22137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:35.899983 22131 server_base.cc:1034] running on GCE node
I20260928 15:05:35.900751 22131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:35.902149 22131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:35.903339 22131 hybrid_clock.cc:648] HybridClock initialized: now 1790607935903284 us; error 50 us; skew 500 ppm
I20260928 15:05:35.903652 22131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:35 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607935, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:35.906670 22131 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:35.908010 22131 webserver.cc:460] Webserver started at http://127.9.12.254:36737/ using document root <none> and password file <none>
I20260928 15:05:35.908689 22131 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:35.908797 22131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:35 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607935, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:35.979269 22131 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 15:05:35.982285 22147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:35.983628 22131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:05:35.983774 22131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "a963e91154a642f1aa3f85bc2b1ea588"
format_stamp: "Formatted at 2026-09-28 15:05:31 on dist-test-slave-dmhs"
server_key: "a864b1bc9fff4a957f04b36ff573540a"
server_key_iv: "af2f2c7635385c35cfb927ea957109b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:35.984752 22131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:36.003587 22131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:36.007555 22131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:36.007799 22131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:36.025754 22131 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:43687
I20260928 15:05:36.025779 22198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:43687 every 8 connection(s)
I20260928 15:05:36.027256 22131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:36.034119 22199 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:05:36.035929 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22131
I20260928 15:05:36.037020 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:36981
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:43687
--webserver_port=39801
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:36.038501 22199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Bootstrap starting.
I20260928 15:05:36.059204 22199 log.cc:824] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:36.068389 22199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:36.068933 22199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Bootstrap complete.
I20260928 15:05:36.077966 22199 raft_consensus.cc:348] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:36.078795 22199 raft_consensus.cc:729] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a963e91154a642f1aa3f85bc2b1ea588, State: Initialized, Role: FOLLOWER
I20260928 15:05:36.079409 22199 consensus_queue.cc:260] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:36.079638 22199 raft_consensus.cc:388] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:36.079746 22199 raft_consensus.cc:482] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:36.079931 22199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:05:36.083585 22199 raft_consensus.cc:504] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:36.084079 22199 leader_election.cc:302] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a963e91154a642f1aa3f85bc2b1ea588; no voters:
I20260928 15:05:36.084728 22199 leader_election.cc:288] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:05:36.084861 22203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:05:36.085954 22203 raft_consensus.cc:686] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [term 2 LEADER]: Becoming Leader. State: Replica: a963e91154a642f1aa3f85bc2b1ea588, State: Running, Role: LEADER
I20260928 15:05:36.086545 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } }
I20260928 15:05:36.086871 22199 sys_catalog.cc:564] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:36.089111 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a963e91154a642f1aa3f85bc2b1ea588. Latest consensus state: current_term: 2 leader_uuid: "a963e91154a642f1aa3f85bc2b1ea588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } } }
I20260928 15:05:36.089538 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:36.090193 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a963e91154a642f1aa3f85bc2b1ea588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a963e91154a642f1aa3f85bc2b1ea588" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43687 } } }
I20260928 15:05:36.091207 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:36.100893 22209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:36.108346 22209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:36.109997 22209 catalog_manager.cc:1259] Loaded cluster ID: 63b5adb21e804973b02ae30e0bb03a31
I20260928 15:05:36.110181 22209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:36.114182 22209 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:36.115754 22209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Loaded TSK: 0
I20260928 15:05:36.117869 22209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:05:36.121320 22199 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 15:05:36.463652 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:36.464063 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:36.464287 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:36.476023 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:36.476225 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:36.476471 22201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:05:36.492915 22201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:36981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=39801
--tserver_master_addrs=127.9.12.254:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:36.494886 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:36.496719 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:36.527467 22237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:36.528651 22238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:36.532843 22201 server_base.cc:1034] running on GCE node
W20260928 15:05:36.533636 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:36.534564 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:36.535851 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:36.537120 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790607936537061 us; error 50 us; skew 500 ppm
I20260928 15:05:36.537597 22201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:36 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:36.547356 22201 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:05:36.549610 22201 webserver.cc:460] Webserver started at http://127.9.12.193:39801/ using document root <none> and password file <none>
I20260928 15:05:36.550602 22201 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:36.550796 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:36 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:36.601244 22201 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.002s sys 0.009s
I20260928 15:05:36.605345 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:36.610128 22201 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.003s
I20260928 15:05:36.610419 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "b8914bb88e7841959a1f5acbe96146a1"
format_stamp: "Formatted at 2026-09-28 15:05:33 on dist-test-slave-dmhs"
server_key: "4550bb78abd4b41d1350ee4a806e0a65"
server_key_iv: "69ead1ff9808fe0275855a262a193a10"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:36.611974 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:36.679044 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:36.684751 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:36.685175 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:36.692822 22201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:36.705336 22201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:36.705576 22201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:05:36.705789 22201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:36.705962 22201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:36.778474 22201 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:36981
I20260928 15:05:36.778537 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:36981 every 8 connection(s)
I20260928 15:05:36.780558 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:36.787746 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22201
I20260928 15:05:36.788604 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:37527
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:43687
--webserver_port=39491
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:36 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:36.825557 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:36.784418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43687 (local address 127.9.12.193:58439)
0928 15:05:36.793993 (+ 9575us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:36.794073 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:36.798913 (+ 4840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:36.802954 (+ 4041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:36.802962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:36.803541 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:36.804172 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:36.804191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:36.806942 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:36.806951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:36.809251 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:36.809277 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:36.811168 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:36.811177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:36.811484 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:36.812317 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:36.812356 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:36.815274 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:36.820134 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:36.820145 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:36.820165 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:36.820845 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:36.821375 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:36.821382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:36.821386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:36.821693 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:36.822740 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:36.822750 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:36.823236 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 15:05:36.823582 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 15:05:36.824211 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9089,"thread_start_us":10993,"threads_started":1}
I20260928 15:05:36.826633 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:36.783997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:58439 (local address 127.9.12.254:43687)
0928 15:05:36.800535 (+ 16538us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:36.800554 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:36.800609 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:05:36.801085 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:36.801109 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:36.801622 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:36.802080 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:36.804505 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:36.806699 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:36.809443 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:36.810589 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:36.810760 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:36.815616 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:36.815628 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:36.815648 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:36.815699 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:36.819939 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:36.821029 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:36.821038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:36.821047 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:36.821241 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:36.821849 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:36.821854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:36.821857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:36.822575 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:36.822808 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:36.823840 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0928 15:05:36.825314 (+ 1474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16191,"thread_start_us":16229,"threads_started":1}
I20260928 15:05:36.830423 22362 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43687
I20260928 15:05:36.830727 22362 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:36.831615 22362 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
W20260928 15:05:36.834581 22362 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:43687 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:05:36.834776 22362 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:36.835034 22362 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:36.836159 22362 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:36.836508 22362 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
W20260928 15:05:36.837656 22362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:05:37.215674 22366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:37.215986 22366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:37.216137 22366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:37.226046 22366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:37.226181 22366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:37.226406 22366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:05:37.240213 22366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:37527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=39491
--tserver_master_addrs=127.9.12.254:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:37.241957 22366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:37.243640 22366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:37.260560 22381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.264761 22383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.265901 22380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:37.278249 22366 server_base.cc:1034] running on GCE node
I20260928 15:05:37.279217 22366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:37.292599 22366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:37.300237 22366 hybrid_clock.cc:648] HybridClock initialized: now 1790607937300134 us; error 104 us; skew 500 ppm
I20260928 15:05:37.300738 22366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:37 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:37.305156 22366 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:05:37.332182 22366 webserver.cc:460] Webserver started at http://127.9.12.194:39491/ using document root <none> and password file <none>
I20260928 15:05:37.333002 22366 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:37.333176 22366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:37 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:37.353863 22366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 15:05:37.356698 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.358175 22366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:05:37.358353 22366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "dda682f8139b40318213750a57ff20a8"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "93fc7e285406f4fe3d6f268e5055f4f0"
server_key_iv: "6659dff5fe6ad27d010029a1b9153097"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.359472 22366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:37.379040 22366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:37.383378 22366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:37.383735 22366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:37.385035 22366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 28 15:05:37 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:37.386719 22366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:37.386838 22366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.386978 22366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:37.387069 22366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.466131 22366 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:37527
I20260928 15:05:37.466183 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:37527 every 8 connection(s)
I20260928 15:05:37.467834 22366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:37.476131 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22366
I20260928 15:05:37.477016 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.12.212:42215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:46367
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:43687
--webserver_port=39995
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:37 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:37.500167 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:37.472058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43687 (local address 127.9.12.194:59797)
0928 15:05:37.476940 (+ 4882us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:37.476989 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:37.479452 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:37.480604 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:37.480617 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:37.481232 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:37.482038 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:37.482063 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.483637 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.483643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:37.485666 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:37.485682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.486783 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.486791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:37.487111 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:37.487920 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:37.487950 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:37.490996 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:37.495146 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:37.495155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:37.495175 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:37.495779 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:37.496221 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:37.496225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:37.496228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:37.496519 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:37.497300 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:37.497309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:37.497781 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 15:05:37.498096 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 15:05:37.498702 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4581,"thread_start_us":4339,"threads_started":1}
I20260928 15:05:37.501296 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:37.471128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:59797 (local address 127.9.12.254:43687)
0928 15:05:37.474716 (+ 3588us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:37.474726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:37.477420 (+ 2694us) server_negotiation.cc:407] Connection header received
0928 15:05:37.479915 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:37.479921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:37.480020 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:37.480126 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:37.482256 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.483442 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.485843 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.486480 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.486692 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:37.491279 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:37.491286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:37.491290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:37.491332 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:37.494976 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:37.495923 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:37.495929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:37.495933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:37.496082 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:37.496674 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:37.496681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:37.496684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:37.497058 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:37.497219 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:37.500705 (+ 3486us) server_negotiation.cc:299] Negotiation successful
0928 15:05:37.500938 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3397,"thread_start_us":3269,"threads_started":1}
I20260928 15:05:37.503636 22507 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43687
I20260928 15:05:37.503965 22507 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:37.504936 22507 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:37.507066 22164 ts_manager.cc:194] Registered new tserver with Master: dda682f8139b40318213750a57ff20a8 (127.9.12.194:37527)
I20260928 15:05:37.509286 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:59797
I20260928 15:05:37.837795 22362 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:37.838265 22362 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:37.839543 22164 ts_manager.cc:194] Registered new tserver with Master: b8914bb88e7841959a1f5acbe96146a1 (127.9.12.193:36981)
I20260928 15:05:37.840699 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:58439
Sep 28 15:05:37 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
W20260928 15:05:37.882014 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:37.882450 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:37.882637 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:37.894198 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:37.894409 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:37.894622 22511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:05:37.910377 22511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.12.212:42215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:46367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=39995
--tserver_master_addrs=127.9.12.254:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:37.912191 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:37.913902 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:37.931661 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.933439 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.940331 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:37.941320 22511 server_base.cc:1034] running on GCE node
I20260928 15:05:37.942118 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:37.943441 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:37.944727 22511 hybrid_clock.cc:648] HybridClock initialized: now 1790607937944656 us; error 59 us; skew 500 ppm
I20260928 15:05:37.945122 22511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:37 dist-test-slave-dmhs krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:37.949200 22511 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:05:37.977798 22511 webserver.cc:460] Webserver started at http://127.9.12.195:39995/ using document root <none> and password file <none>
I20260928 15:05:37.978969 22511 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:37.979209 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:38 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:38.049604 22511 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.009s
I20260928 15:05:38.053954 22527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.055755 22511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 15:05:38.055935 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "a875d781ef1c4cf5a6b41a117afb253a"
format_stamp: "Formatted at 2026-09-28 15:05:34 on dist-test-slave-dmhs"
server_key: "cee869d0cff4d029f9ef94a69fceee15"
server_key_iv: "0b6bb3ec105422130670d21987162296"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.057315 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:38.080644 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:38.085752 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:38.086048 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:38.087244 22511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:38.089461 22511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:38.089582 22511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.089685 22511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:38.089761 22511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.142823 22511 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:46367
I20260928 15:05:38.142881 22639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:46367 every 8 connection(s)
I20260928 15:05:38.144704 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:38.146885 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22511
Sep 28 15:05:38 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:38.188220 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.148730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.9.12.254:43687)
0928 15:05:38.158874 (+ 10144us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.158879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.158899 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:05:38.158993 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.158998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.159103 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.159225 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.164038 (+ 4813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.165591 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.171020 (+ 5429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.171705 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.171826 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.175771 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.175777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.175782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.175837 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.181967 (+ 6130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.184259 (+ 2292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.184266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.184269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.184481 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.184952 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.184969 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.184973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.185360 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.187343 (+ 1983us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.187487 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.187790 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10013,"thread_start_us":9893,"threads_started":1}
I20260928 15:05:38.191717 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.160038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43687 (local address 127.9.12.195:42353)
0928 15:05:38.169219 (+ 9181us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.169282 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.171835 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.173081 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.173093 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.173759 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:38.174589 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.174615 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.176487 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.176493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.177889 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.177918 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.179130 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.179141 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.179486 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.180209 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:38.180244 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:38.183141 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:38.186826 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.186835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.186876 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.187493 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.187922 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.187926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.187930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.188221 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.189011 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:38.189019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:38.189523 (+ 504us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.189824 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.190369 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8849,"thread_start_us":8595,"threads_started":1}
I20260928 15:05:38.192800 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.159137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:42353 (local address 127.9.12.254:43687)
0928 15:05:38.168512 (+ 9375us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.168518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.172286 (+ 3768us) server_negotiation.cc:407] Connection header received
0928 15:05:38.172348 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.172353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.172457 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.172564 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.174821 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.176259 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.178098 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.178848 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.179025 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.183446 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.183453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.183458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.183495 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.186663 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.187634 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.187639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.187642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.187790 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.188447 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.188452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.188455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.188795 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.188933 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.192154 (+ 3221us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.192410 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9229,"thread_start_us":9380,"threads_started":1}
I20260928 15:05:38.194203 22640 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43687
I20260928 15:05:38.194543 22640 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.195425 22640 heartbeater.cc:511] Master 127.9.12.254:43687 requested a full tablet report, sending...
I20260928 15:05:38.197182 22164 ts_manager.cc:194] Registered new tserver with Master: a875d781ef1c4cf5a6b41a117afb253a (127.9.12.195:46367)
I20260928 15:05:38.198326 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:42353
I20260928 15:05:38.204057 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:38.225957 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.207721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.9.12.254:43687)
0928 15:05:38.207911 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.207918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.208811 (+ 893us) server_negotiation.cc:407] Connection header received
0928 15:05:38.209096 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.209102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.209185 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.209253 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.211522 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.212679 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.214710 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.215337 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.215451 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.219273 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.219280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.219285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.219330 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.222772 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.223606 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.223614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.223618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.223795 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.224223 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.224229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.224233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.224660 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.224921 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.225341 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.225578 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:05:38 dist-test-slave-dmhs krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for HTTP/127.9.12.212@KRBTEST.COM
I20260928 15:05:38.231796 22164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:05:38.244733 22164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:05:38.393647 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.323745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36981 (local address 127.0.0.1:53090)
0928 15:05:38.381698 (+ 57953us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.381712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.381872 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.382914 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.382921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.382966 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:38.383503 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.383522 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.385678 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.385684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.388338 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.388354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.392582 (+ 4228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.392588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.392780 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.392835 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.392975 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.393179 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57865,"mutex_wait_us":46,"thread_start_us":55971,"threads_started":1}
I20260928 15:05:38.394471 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.324295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.9.12.193:36981)
0928 15:05:38.378878 (+ 54583us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.378887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.381838 (+ 2951us) server_negotiation.cc:407] Connection header received
0928 15:05:38.382060 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.382068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.382369 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.382596 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:38.383765 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.385499 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.388542 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.390052 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.390175 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.390436 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.393901 (+ 3465us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.394097 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54402,"thread_start_us":55207,"threads_started":1}
I20260928 15:05:38.397902 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.315662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:46367 (local address 127.0.0.1:48824)
0928 15:05:38.379278 (+ 63616us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.379298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.379429 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.383269 (+ 3840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.383275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.383296 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:38.383721 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.383733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.385678 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.385684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.392339 (+ 6655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.392357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.396992 (+ 4635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.396998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.397181 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.397218 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.397316 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.397497 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63497,"mutex_wait_us":3,"thread_start_us":64632,"threads_started":1}
I20260928 15:05:38.398787 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.316387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.9.12.195:46367)
0928 15:05:38.378402 (+ 62015us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.378412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.382267 (+ 3855us) server_negotiation.cc:407] Connection header received
0928 15:05:38.382379 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.382386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.382735 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.382936 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:38.383909 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.385530 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.395383 (+ 9853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.396767 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.396937 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.397192 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.398230 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.398405 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61816,"thread_start_us":64192,"threads_started":1}
I20260928 15:05:38.399516 22297 tablet_service.cc:1467] Processing CreateTablet for tablet 1a628b159da847f5ad09ebdd31cada75 (DEFAULT_TABLE table=test-table [id=08cee89ec2fb41948726d0529bb413be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:38.401166 22297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a628b159da847f5ad09ebdd31cada75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:38.402621 22575 tablet_service.cc:1467] Processing CreateTablet for tablet 1a628b159da847f5ad09ebdd31cada75 (DEFAULT_TABLE table=test-table [id=08cee89ec2fb41948726d0529bb413be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:38.403909 22575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a628b159da847f5ad09ebdd31cada75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:38.414821 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.312615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:37527 (local address 127.0.0.1:49094)
0928 15:05:38.377375 (+ 64760us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.377444 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.377683 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.389612 (+ 11929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.389616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.389659 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:38.390018 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.390030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.397373 (+ 7343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.397378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.411880 (+ 14502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.411902 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.413936 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.413956 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.414156 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.414191 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.414286 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.414447 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64397,"thread_start_us":69454,"threads_started":1}
I20260928 15:05:38.417536 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.312784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.9.12.194:37527)
0928 15:05:38.388934 (+ 76150us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.388942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.388988 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:05:38.389091 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.389098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.389398 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.391662 (+ 2264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:38.391874 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.393401 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.412190 (+ 18789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.413682 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.413859 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.414099 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.416645 (+ 2546us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.417282 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75971,"thread_start_us":79495,"threads_started":1}
I20260928 15:05:38.429840 22441 tablet_service.cc:1467] Processing CreateTablet for tablet 1a628b159da847f5ad09ebdd31cada75 (DEFAULT_TABLE table=test-table [id=08cee89ec2fb41948726d0529bb413be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:38.432126 22441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a628b159da847f5ad09ebdd31cada75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:38.442601 22663 tablet_bootstrap.cc:492] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Bootstrap starting.
I20260928 15:05:38.443007 22664 tablet_bootstrap.cc:492] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Bootstrap starting.
I20260928 15:05:38.448180 22663 tablet_bootstrap.cc:654] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:38.448400 22664 tablet_bootstrap.cc:654] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:38.449947 22663 log.cc:824] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:38.461773 22664 log.cc:824] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:38.462792 22663 tablet_bootstrap.cc:492] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: No bootstrap required, opened a new log
I20260928 15:05:38.463256 22663 ts_tablet_manager.cc:1397] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.015s
I20260928 15:05:38.479254 22667 tablet_bootstrap.cc:492] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Bootstrap starting.
I20260928 15:05:38.479950 22663 raft_consensus.cc:348] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.480414 22663 raft_consensus.cc:374] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:38.480530 22663 raft_consensus.cc:729] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a875d781ef1c4cf5a6b41a117afb253a, State: Initialized, Role: FOLLOWER
I20260928 15:05:38.481164 22663 consensus_queue.cc:260] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.481844 22664 tablet_bootstrap.cc:492] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: No bootstrap required, opened a new log
I20260928 15:05:38.482204 22664 ts_tablet_manager.cc:1397] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.005s
I20260928 15:05:38.482299 22640 heartbeater.cc:503] Master 127.9.12.254:43687 was elected leader, sending a full tablet report...
I20260928 15:05:38.484189 22663 ts_tablet_manager.cc:1428] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Time spent starting tablet: real 0.021s user 0.016s sys 0.000s
I20260928 15:05:38.492043 22667 tablet_bootstrap.cc:654] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:38.494146 22667 log.cc:824] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:38.494935 22664 raft_consensus.cc:348] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.495569 22664 raft_consensus.cc:374] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:38.495715 22664 raft_consensus.cc:729] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8914bb88e7841959a1f5acbe96146a1, State: Initialized, Role: FOLLOWER
I20260928 15:05:38.496560 22664 consensus_queue.cc:260] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.501834 22664 ts_tablet_manager.cc:1428] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260928 15:05:38.502250 22362 heartbeater.cc:503] Master 127.9.12.254:43687 was elected leader, sending a full tablet report...
I20260928 15:05:38.510620 22667 tablet_bootstrap.cc:492] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: No bootstrap required, opened a new log
I20260928 15:05:38.510954 22667 ts_tablet_manager.cc:1397] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.008s
I20260928 15:05:38.514310 22507 heartbeater.cc:503] Master 127.9.12.254:43687 was elected leader, sending a full tablet report...
I20260928 15:05:38.524909 22667 raft_consensus.cc:348] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.525579 22667 raft_consensus.cc:374] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:38.525736 22667 raft_consensus.cc:729] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda682f8139b40318213750a57ff20a8, State: Initialized, Role: FOLLOWER
I20260928 15:05:38.526727 22667 consensus_queue.cc:260] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.529073 22667 ts_tablet_manager.cc:1428] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
W20260928 15:05:38.552753 22363 tablet.cc:2369] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:05:38.657744 22641 tablet.cc:2369] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:38.709901 22668 raft_consensus.cc:482] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:38.710234 22668 raft_consensus.cc:504] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.711787 22668 leader_election.cc:288] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda682f8139b40318213750a57ff20a8 (127.9.12.194:37527), b8914bb88e7841959a1f5acbe96146a1 (127.9.12.193:36981)
I20260928 15:05:38.719368 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.712239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:42807 (local address 127.9.12.193:36981)
0928 15:05:38.712489 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.712495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.712727 (+ 232us) server_negotiation.cc:407] Connection header received
0928 15:05:38.712969 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.712974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.713076 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.713185 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:38.713758 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.714753 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.717213 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.718417 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.718587 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.718663 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.718982 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.719128 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:05:38.719551 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.712085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:36981 (local address 127.9.12.195:42807)
0928 15:05:38.712610 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.712630 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.712820 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.713188 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.713191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.713210 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:38.713577 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.713588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.714874 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.714877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.717064 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.717079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.718611 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.718615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.718764 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.718830 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.718941 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.719079 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":82,"thread_start_us":336,"threads_started":1}
I20260928 15:05:38.719955 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.712259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:52283 (local address 127.9.12.194:37527)
0928 15:05:38.712500 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.712507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.713032 (+ 525us) server_negotiation.cc:407] Connection header received
0928 15:05:38.713146 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.713155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.713241 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.713331 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:38.714156 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.715062 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.717416 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.718662 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.718785 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.718853 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.719240 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.719356 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:05:38.719974 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.712085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:37527 (local address 127.9.12.195:52283)
0928 15:05:38.712929 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.712944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.713047 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.713352 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.713356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.713381 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:38.713979 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.713989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.715226 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.715232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.717295 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.717305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.718867 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.718874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.719070 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.719096 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.719185 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.719307 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":170,"thread_start_us":364,"threads_started":1}
I20260928 15:05:38.720626 22317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a628b159da847f5ad09ebdd31cada75" candidate_uuid: "a875d781ef1c4cf5a6b41a117afb253a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8914bb88e7841959a1f5acbe96146a1" is_pre_election: true
I20260928 15:05:38.720948 22461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a628b159da847f5ad09ebdd31cada75" candidate_uuid: "a875d781ef1c4cf5a6b41a117afb253a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda682f8139b40318213750a57ff20a8" is_pre_election: true
I20260928 15:05:38.721261 22317 raft_consensus.cc:2454] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a875d781ef1c4cf5a6b41a117afb253a in term 0.
I20260928 15:05:38.721799 22461 raft_consensus.cc:2454] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a875d781ef1c4cf5a6b41a117afb253a in term 0.
I20260928 15:05:38.722067 22530 leader_election.cc:302] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a875d781ef1c4cf5a6b41a117afb253a, b8914bb88e7841959a1f5acbe96146a1; no voters:
I20260928 15:05:38.722496 22668 raft_consensus.cc:2781] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:38.722695 22668 raft_consensus.cc:482] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:38.722813 22668 raft_consensus.cc:3037] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:38.724906 22668 raft_consensus.cc:504] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.725481 22668 leader_election.cc:288] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [CANDIDATE]: Term 1 election: Requested vote from peers dda682f8139b40318213750a57ff20a8 (127.9.12.194:37527), b8914bb88e7841959a1f5acbe96146a1 (127.9.12.193:36981)
I20260928 15:05:38.725960 22461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a628b159da847f5ad09ebdd31cada75" candidate_uuid: "a875d781ef1c4cf5a6b41a117afb253a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda682f8139b40318213750a57ff20a8"
I20260928 15:05:38.726018 22317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a628b159da847f5ad09ebdd31cada75" candidate_uuid: "a875d781ef1c4cf5a6b41a117afb253a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8914bb88e7841959a1f5acbe96146a1"
I20260928 15:05:38.726158 22461 raft_consensus.cc:3037] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:38.726297 22317 raft_consensus.cc:3037] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:05:38.726953 22508 tablet.cc:2369] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:38.729341 22461 raft_consensus.cc:2454] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a875d781ef1c4cf5a6b41a117afb253a in term 1.
I20260928 15:05:38.729694 22528 leader_election.cc:302] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a875d781ef1c4cf5a6b41a117afb253a, dda682f8139b40318213750a57ff20a8; no voters:
I20260928 15:05:38.729939 22668 raft_consensus.cc:2781] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:38.730248 22317 raft_consensus.cc:2454] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a875d781ef1c4cf5a6b41a117afb253a in term 1.
I20260928 15:05:38.730993 22668 raft_consensus.cc:686] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 1 LEADER]: Becoming Leader. State: Replica: a875d781ef1c4cf5a6b41a117afb253a, State: Running, Role: LEADER
I20260928 15:05:38.731662 22668 consensus_queue.cc:237] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } }
I20260928 15:05:38.736215 22164 catalog_manager.cc:5495] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a reported cstate change: term changed from 0 to 1, leader changed from <none> to a875d781ef1c4cf5a6b41a117afb253a (127.9.12.195). New cstate: current_term: 1 leader_uuid: "a875d781ef1c4cf5a6b41a117afb253a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a875d781ef1c4cf5a6b41a117afb253a" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 46367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 } health_report { overall_health: UNKNOWN } } }
I20260928 15:05:38.778592 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.773194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.9.12.195:46367)
0928 15:05:38.773432 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.773451 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.773758 (+ 307us) server_negotiation.cc:407] Connection header received
0928 15:05:38.773929 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.773932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.774046 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.774165 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:38.774492 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.775206 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.776561 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.776934 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.777024 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.777470 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:38.778043 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:38.778150 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.778273 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.778336 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:05:38.787424 22461 raft_consensus.cc:1261] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 1 FOLLOWER]: Refusing update from remote peer a875d781ef1c4cf5a6b41a117afb253a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:38.787429 22317 raft_consensus.cc:1261] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 1 FOLLOWER]: Refusing update from remote peer a875d781ef1c4cf5a6b41a117afb253a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:38.788672 22681 consensus_queue.cc:1035] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8914bb88e7841959a1f5acbe96146a1" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:38.789062 22668 consensus_queue.cc:1035] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda682f8139b40318213750a57ff20a8" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:38.800246 22685 mvcc.cc:205] Tried to move back new op lower bound from 7334330117261979648 to 7334330117049483264. Current Snapshot: MvccSnapshot[applied={T|T < 7334330117261979648}]
I20260928 15:05:38.800734 22687 mvcc.cc:205] Tried to move back new op lower bound from 7334330117261979648 to 7334330117049483264. Current Snapshot: MvccSnapshot[applied={T|T < 7334330117261979648}]
I20260928 15:05:38.801147 22686 mvcc.cc:205] Tried to move back new op lower bound from 7334330117261979648 to 7334330117049483264. Current Snapshot: MvccSnapshot[applied={T|T < 7334330117261979648}]
I20260928 15:05:38.814949 22164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:38.815158 22164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:38.818403 22164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a963e91154a642f1aa3f85bc2b1ea588: Sending DeleteTablet for 3 replicas of tablet 1a628b159da847f5ad09ebdd31cada75
I20260928 15:05:38.819497 22441 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a628b159da847f5ad09ebdd31cada75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:38 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:49094
I20260928 15:05:38.819510 22575 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a628b159da847f5ad09ebdd31cada75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:38 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48824
I20260928 15:05:38.819830 22297 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a628b159da847f5ad09ebdd31cada75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:38 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:53090
I20260928 15:05:38.823037 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22201
I20260928 15:05:38.833388 22695 tablet_replica.cc:330] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: stopping tablet replica
I20260928 15:05:38.834054 22697 tablet_replica.cc:330] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: stopping tablet replica
I20260928 15:05:38.834558 22697 raft_consensus.cc:2229] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:38.834613 22695 raft_consensus.cc:2229] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:38.835140 22697 raft_consensus.cc:2258] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:38.835283 22695 raft_consensus.cc:2258] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:38.837381 22695 ts_tablet_manager.cc:1905] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:38.837404 22697 ts_tablet_manager.cc:1905] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:38.844182 22695 ts_tablet_manager.cc:1918] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:38.844400 22695 log.cc:1196] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/1a628b159da847f5ad09ebdd31cada75
I20260928 15:05:38.844980 22695 ts_tablet_manager.cc:1939] T 1a628b159da847f5ad09ebdd31cada75 P dda682f8139b40318213750a57ff20a8: Deleting consensus metadata
I20260928 15:05:38.845403 22697 ts_tablet_manager.cc:1918] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:38.845562 22697 log.cc:1196] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/1a628b159da847f5ad09ebdd31cada75
I20260928 15:05:38.846050 22697 ts_tablet_manager.cc:1939] T 1a628b159da847f5ad09ebdd31cada75 P b8914bb88e7841959a1f5acbe96146a1: Deleting consensus metadata
I20260928 15:05:38.847102 22148 catalog_manager.cc:4841] TS dda682f8139b40318213750a57ff20a8 (127.9.12.194:37527): tablet 1a628b159da847f5ad09ebdd31cada75 (table test-table [id=08cee89ec2fb41948726d0529bb413be]) successfully deleted
I20260928 15:05:38.848366 22150 catalog_manager.cc:4841] TS b8914bb88e7841959a1f5acbe96146a1 (127.9.12.193:36981): tablet 1a628b159da847f5ad09ebdd31cada75 (table test-table [id=08cee89ec2fb41948726d0529bb413be]) successfully deleted
I20260928 15:05:38.851797 22696 tablet_replica.cc:330] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: stopping tablet replica
I20260928 15:05:38.852202 22696 raft_consensus.cc:2229] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:05:38.853078 22696 raft_consensus.cc:2258] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:38.855355 22696 ts_tablet_manager.cc:1905] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:38.861950 22696 ts_tablet_manager.cc:1918] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:38.862110 22696 log.cc:1196] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/1a628b159da847f5ad09ebdd31cada75
I20260928 15:05:38.862679 22696 ts_tablet_manager.cc:1939] T 1a628b159da847f5ad09ebdd31cada75 P a875d781ef1c4cf5a6b41a117afb253a: Deleting consensus metadata
I20260928 15:05:38.864481 22151 catalog_manager.cc:4841] TS a875d781ef1c4cf5a6b41a117afb253a (127.9.12.195:46367): tablet 1a628b159da847f5ad09ebdd31cada75 (table test-table [id=08cee89ec2fb41948726d0529bb413be]) successfully deleted
I20260928 15:05:38.872656 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.824142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.9.12.193:36981)
0928 15:05:38.824442 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.824452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.845923 (+ 21471us) server_negotiation.cc:407] Connection header received
0928 15:05:38.846180 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.846187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.846263 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.846353 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.848061 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.849197 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.851464 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.852064 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.852185 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.855700 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.855712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.855735 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.855790 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.859767 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.870149 (+ 10382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.870160 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.870163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.870352 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.870858 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.870863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.870866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.871370 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.871711 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.871948 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.872199 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:05:38.873383 22357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:38.959501 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22201
I20260928 15:05:38.981644 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22366
I20260928 15:05:38.997063 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.982462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.9.12.194:37527)
0928 15:05:38.982627 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.982631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.982648 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:05:38.982836 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.982840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.982916 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.983004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.984784 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.985546 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.987379 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.987982 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.988107 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.991282 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.991296 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.991311 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.991344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.994138 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.994966 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.994973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.994975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.995103 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.995507 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.995510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.995512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.995826 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.996073 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.996536 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.996693 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:05:38.997885 22501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:39.084016 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22366
I20260928 15:05:39.100812 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22511
I20260928 15:05:39.116305 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:39.101596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.9.12.195:46367)
0928 15:05:39.101753 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:39.101757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:39.101806 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:05:39.102022 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:39.102026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:39.102094 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:39.102176 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:39.104150 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.105119 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.106390 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.106956 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.107091 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:39.110376 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:39.110388 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:39.110403 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:39.110437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:39.113416 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:39.114257 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:39.114264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:39.114266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:39.114430 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:39.114856 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:39.114859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:39.114862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:39.115210 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:39.115421 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:39.115786 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 15:05:39.115935 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:05:39.117230 22635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:39.234856 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22511
I20260928 15:05:39.254005 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22131
I20260928 15:05:39.255118 22194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:39.364356 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22131
I20260928 15:05:39.388033 9267 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:05:39.722816 9267 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:05:39.723060 9267 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:05:40.068310 9267 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:05:40.068 UTC [21183] LOG: received smart shutdown request
2026-09-28 15:05:40.071 UTC [21183] LOG: background worker "logical replication launcher" (PID 21192) exited with exit code 1
2026-09-28 15:05:40.072 UTC [21187] LOG: shutting down
2026-09-28 15:05:40.233 UTC [21183] LOG: database system is shut down
2026-09-28T15:05:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52601 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:05:40 dist-test-slave-dmhs krb5kdc[22711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:05:40 dist-test-slave-dmhs krb5kdc[22711](info): set up 2 sockets
Sep 28 15:05:40 dist-test-slave-dmhs krb5kdc[22711](info): commencing operation
krb5kdc: starting...
W20260928 15:05:41.456825 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:05:41 dist-test-slave-dmhs krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:05:41.703536 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42405
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42405,127.9.12.253:34847,127.9.12.252:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:42.033844 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:42.034142 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:42.034307 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:42.041570 22724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:42.041708 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:42.041807 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:42.041901 22724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:42.041993 22724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:42.052007 22724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42405,127.9.12.253:34847,127.9.12.252:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:42.053407 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:42.054724 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:42.066298 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:42.066361 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:42.066587 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:42.067049 22724 server_base.cc:1034] running on GCE node
I20260928 15:05:42.067806 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:42.069253 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:42.070453 22724 hybrid_clock.cc:648] HybridClock initialized: now 1790607942070388 us; error 57 us; skew 500 ppm
I20260928 15:05:42.070807 22724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:42 dist-test-slave-dmhs krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:42.080498 22724 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:42.082145 22724 webserver.cc:460] Webserver started at http://127.9.12.254:34455/ using document root <none> and password file <none>
I20260928 15:05:42.082842 22724 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:42.082979 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:42.083293 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:42.086472 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "9573e628338d451e83525719fb21f514"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-dmhs"
server_key: "263e2726749d3b1e1342458e3ff2ff53"
server_key_iv: "f505fe89568c2697805d01ed2dd94e7d"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.087316 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "9573e628338d451e83525719fb21f514"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-dmhs"
server_key: "263e2726749d3b1e1342458e3ff2ff53"
server_key_iv: "f505fe89568c2697805d01ed2dd94e7d"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.092650 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:05:42.096088 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:42.097481 22724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:42.097651 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "9573e628338d451e83525719fb21f514"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-dmhs"
server_key: "263e2726749d3b1e1342458e3ff2ff53"
server_key_iv: "f505fe89568c2697805d01ed2dd94e7d"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.097860 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:42.114812 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:42.118913 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:42.119221 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:42.137898 22724 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:42405
I20260928 15:05:42.137933 22790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:42405 every 8 connection(s)
I20260928 15:05:42.139330 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:42.143033 22791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:42.145471 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22724
I20260928 15:05:42.145779 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:05:42.146437 9267 external_mini_cluster.cc:1428] Setting key 0c140d0c5eb7113439686fa415d8d579
I20260928 15:05:42.149899 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:05:42 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.253@KRBTEST.COM" created.
I20260928 15:05:42.172152 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.153151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42405 (local address 127.0.0.1:34696)
0928 15:05:42.154102 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.154154 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:42.156125 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:42.157029 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:42.157036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:42.157766 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:42.158114 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.158132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.159666 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.159671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.160814 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.160827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.161740 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.161747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.161910 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.162620 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:42.162653 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:42.164777 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:42.167894 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.167906 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.167917 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.168369 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.168761 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.168767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.168771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.168925 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.169689 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:42.169703 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:42.169920 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 15:05:42.170166 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 15:05:42.170780 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":139,"thread_start_us":425,"threads_started":1}
I20260928 15:05:42.172173 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.153153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.9.12.254:42405)
0928 15:05:42.154918 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:42.154926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:42.154976 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:05:42.156540 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:42.156546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:42.156797 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:42.157001 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:42.158305 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.159510 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.160951 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.161504 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.161704 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.164988 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:42.164995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:42.165006 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:42.165034 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:42.167769 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.168505 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.168512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.168514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.168654 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.169059 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.169068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.169071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.169558 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:42.169772 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:42.170291 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 15:05:42.170680 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":1547,"thread_start_us":1328,"threads_started":1}
I20260928 15:05:42.175400 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:42.176672 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.176257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55422)
0928 15:05:42.176444 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.176557 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:05:42.176980 22743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
I20260928 15:05:42.178529 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.178182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55432)
0928 15:05:42.178330 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.178391 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:05:42.178725 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:34847: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab.
Entry for principal kudu/127.9.12.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab.
I20260928 15:05:42.195899 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:34847
--webserver_interface=127.9.12.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab
--principal=kudu/127.9.12.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42405,127.9.12.253:34847,127.9.12.252:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:42.228065 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } attempt: 1
I20260928 15:05:42.229234 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.228794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55446)
0928 15:05:42.228993 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.229109 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:05:42.230333 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.229970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55458)
0928 15:05:42.230108 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.230180 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 15:05:42.230513 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:34847: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
I20260928 15:05:42.298771 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } attempt: 2
I20260928 15:05:42.299903 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.299460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55468)
0928 15:05:42.299667 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.299763 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 15:05:42.300982 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.300615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55476)
0928 15:05:42.300750 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.300850 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:05:42.301195 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:34847: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
I20260928 15:05:42.392521 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } attempt: 3
I20260928 15:05:42.393908 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.393291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55484)
0928 15:05:42.393583 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.393734 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 15:05:42.395083 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.394656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55498)
0928 15:05:42.394829 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.394937 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:05:42.395367 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:34847: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
W20260928 15:05:42.521409 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:42.521721 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:42.521871 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:42.529503 22798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:42.529640 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:42.529760 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:42.529855 22798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:42.529953 22798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:42.538600 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } attempt: 4
I20260928 15:05:42.539767 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.539246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55502)
0928 15:05:42.539550 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.539638 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260928 15:05:42.539845 22798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42405,127.9.12.253:34847,127.9.12.252:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab
--principal=kudu/127.9.12.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.12.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:42.540824 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.540450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55512)
0928 15:05:42.540610 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.540708 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:05:42.541000 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:34847: Network error: Client connection negotiation failed: client connection to 127.9.12.253:34847: connect: Connection refused (error 111)
I20260928 15:05:42.541347 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:42.542575 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:42.553879 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:42.553903 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:42.553930 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:42.554687 22798 server_base.cc:1034] running on GCE node
I20260928 15:05:42.555387 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:42.556649 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:42.557832 22798 hybrid_clock.cc:648] HybridClock initialized: now 1790607942557777 us; error 44 us; skew 500 ppm
I20260928 15:05:42.558173 22798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:42 dist-test-slave-dmhs krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:42.561164 22798 init.cc:377] Logged in from keytab as kudu/127.9.12.253@KRBTEST.COM (short username kudu)
I20260928 15:05:42.562520 22798 webserver.cc:460] Webserver started at http://127.9.12.253:38637/ using document root <none> and password file <none>
I20260928 15:05:42.563114 22798 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:42.563205 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:42.563445 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:42.566159 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data/instance:
uuid: "252fa07124384258ab49f68c77547207"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-dmhs"
server_key: "d68f0ef9a52ffa467e4c8a45bb4c0693"
server_key_iv: "a311a16480066023a4d29d4dd1dcfe84"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.566820 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal/instance:
uuid: "252fa07124384258ab49f68c77547207"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-dmhs"
server_key: "d68f0ef9a52ffa467e4c8a45bb4c0693"
server_key_iv: "a311a16480066023a4d29d4dd1dcfe84"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.571638 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 15:05:42.574657 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:42.575842 22798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:05:42.575973 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal
uuid: "252fa07124384258ab49f68c77547207"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-dmhs"
server_key: "d68f0ef9a52ffa467e4c8a45bb4c0693"
server_key_iv: "a311a16480066023a4d29d4dd1dcfe84"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.576121 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:42.594781 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:42.598814 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:42.599056 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:42.616521 22798 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.253:34847
I20260928 15:05:42.616559 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.253:34847 every 8 connection(s)
I20260928 15:05:42.617763 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
I20260928 15:05:42.618888 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22798
I20260928 15:05:42.619086 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-1/wal/instance
I20260928 15:05:42.619648 9267 external_mini_cluster.cc:1428] Setting key fca524d38f05d06c5466a06f91662cb9
I20260928 15:05:42.622202 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:42.631171 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.12.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.252@KRBTEST.COM" created.
Sep 28 15:05:42 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:42.652474 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.633896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.9.12.254:42405)
0928 15:05:42.634075 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:42.634079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:42.635563 (+ 1484us) server_negotiation.cc:407] Connection header received
0928 15:05:42.637567 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:42.637573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:42.637680 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:42.637791 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:42.639351 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.640021 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.641669 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.642119 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.642250 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.646427 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:42.646432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:42.646435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:42.646471 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:42.649119 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.650089 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.650093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.650095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.650211 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.650669 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.650672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.650673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.650936 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:42.651065 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:42.651968 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 15:05:42.652151 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:05:42.653524 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.634499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42405 (local address 127.0.0.1:34698)
0928 15:05:42.635389 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.635432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:42.637270 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:42.638060 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:42.638069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:42.638606 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:42.639180 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.639198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.640191 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.640195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.641507 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.641521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.642355 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.642373 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.642605 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.643334 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:42.643371 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:42.646198 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:42.649277 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.649288 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.649308 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.649973 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.650312 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.650320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.650326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.650560 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.651078 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:42.651088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:42.651562 (+ 474us) client_negotiation.cc:769] Sending connection context
0928 15:05:42.651872 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 15:05:42.652443 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":412,"threads_started":1}
I20260928 15:05:42.655211 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:05:42 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:05:42.671291 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.656061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55522)
0928 15:05:42.656237 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.656258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:42.656432 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:42.657548 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:42.657551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:42.657923 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:42.658064 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.658071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.660126 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.660130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.661028 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.661039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.662379 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.662388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.662539 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.663158 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:42.663188 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:42.665257 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:42.668857 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.668861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.668863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.669337 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.669703 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.669710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.669716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.669915 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.670499 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:42.670507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:42.670636 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 15:05:42.670740 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:05:42.670951 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:05:42.671509 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.656196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.9.12.253:34847)
0928 15:05:42.657043 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:42.657052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:42.657089 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:05:42.657143 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:42.657154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:42.657383 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:42.657527 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:42.658212 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.659985 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.661164 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.662147 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.662298 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.665476 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:42.665491 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:42.665508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:42.665559 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:42.668741 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.669462 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.669468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.669471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.669588 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.670030 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.670033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.670035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.670369 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:42.670559 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:42.670912 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:05:42.671155 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":485,"threads_started":1}
I20260928 15:05:42.673962 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab.
Entry for principal kudu/127.9.12.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab.
I20260928 15:05:42.675050 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.674529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54818)
0928 15:05:42.674683 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.674843 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:05:42.675279 22817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.676486 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:35957
--webserver_interface=127.9.12.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab
--principal=kudu/127.9.12.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42405,127.9.12.253:34847,127.9.12.252:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:42.676988 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.676612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54834)
0928 15:05:42.676824 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.676894 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 15:05:42.677241 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.726554 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 1
I20260928 15:05:42.727813 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.727343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54848)
0928 15:05:42.727566 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.727668 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 15:05:42.729135 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.728807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54862)
0928 15:05:42.728970 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.729037 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 15:05:42.729417 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.797618 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 2
I20260928 15:05:42.798820 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.798379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54868)
0928 15:05:42.798575 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.798665 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:05:42.799782 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.799458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54880)
0928 15:05:42.799600 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.799674 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:05:42.799981 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.840317 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } attempt: 5
Sep 28 15:05:42 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:05:42.855633 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.841095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55536)
0928 15:05:42.841319 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.841341 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:42.841487 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:42.841849 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:42.841855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:42.842398 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:42.842622 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.842635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.843634 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.843639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.845091 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:42.845110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.845968 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.845974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:42.846194 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.846865 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:42.846908 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:42.849869 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:42.853448 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.853452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.853454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.853812 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.854201 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:42.854204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:42.854206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:42.854327 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:42.854919 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:42.854924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:42.855086 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 15:05:42.855196 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:05:42.855390 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:05:42.855894 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.841209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.9.12.253:34847)
0928 15:05:42.841375 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:42.841379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:42.841401 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:05:42.841617 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:42.841621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:42.841697 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:42.841779 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:42.842805 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.843495 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.845252 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:42.845808 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:42.845908 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:42.850066 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:42.850071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:42.850074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:42.850108 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:42.853285 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.853932 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.853937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.853941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.854094 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:42.854423 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:42.854428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:42.854431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:42.854789 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:42.854934 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:42.855322 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 15:05:42.855554 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:05:42.856936 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:42.857955 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.857577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54884)
0928 15:05:42.857730 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.857805 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 15:05:42.858915 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.858605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54886)
0928 15:05:42.858741 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.858823 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:05:42.859123 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.891199 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 3
I20260928 15:05:42.892426 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.891878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54888)
0928 15:05:42.892101 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.892209 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 15:05:42.893503 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.893082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54892)
0928 15:05:42.893307 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.893392 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260928 15:05:42.893709 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.910373 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 1
I20260928 15:05:42.911568 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.911079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54898)
0928 15:05:42.911277 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.911394 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:05:42.912657 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.912335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54914)
0928 15:05:42.912476 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.912549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:05:42.912884 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:42.981087 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 2
I20260928 15:05:42.982144 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.981695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54928)
0928 15:05:42.981900 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.981995 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:05:42.983165 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:42.982823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54936)
0928 15:05:42.982979 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:42.983044 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:05:42.983366 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
W20260928 15:05:43.006968 22872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:43.007325 22872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:43.007483 22872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:43.014520 22872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:43.014672 22872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:43.014775 22872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:43.014875 22872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:43.014958 22872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:43.024793 22872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42405,127.9.12.253:34847,127.9.12.252:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab
--principal=kudu/127.9.12.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.12.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:43.026160 22872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:43.027483 22872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:05:43.036909 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 4
I20260928 15:05:43.038070 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.037560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54942)
0928 15:05:43.037822 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.037912 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:05:43.039263 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.038784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54958)
0928 15:05:43.039013 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.039112 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260928 15:05:43.039603 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
W20260928 15:05:43.040452 22877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:43.040598 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:43.040625 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:43.041920 22872 server_base.cc:1034] running on GCE node
I20260928 15:05:43.042645 22872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:43.044100 22872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:43.045305 22872 hybrid_clock.cc:648] HybridClock initialized: now 1790607943045232 us; error 66 us; skew 500 ppm
I20260928 15:05:43.045680 22872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:43.048897 22872 init.cc:377] Logged in from keytab as kudu/127.9.12.252@KRBTEST.COM (short username kudu)
I20260928 15:05:43.050285 22872 webserver.cc:460] Webserver started at http://127.9.12.252:39897/ using document root <none> and password file <none>
I20260928 15:05:43.050935 22872 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:43.051039 22872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:43.051291 22872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:43.054236 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data/instance:
uuid: "ba196e9839eb47d881e848d10a0455ac"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-dmhs"
server_key: "308b04ce76df6fc1d257ccc5e39d2a5c"
server_key_iv: "db8e41a0034df6fc27f6a5086cde2788"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.055002 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal/instance:
uuid: "ba196e9839eb47d881e848d10a0455ac"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-dmhs"
server_key: "308b04ce76df6fc1d257ccc5e39d2a5c"
server_key_iv: "db8e41a0034df6fc27f6a5086cde2788"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.060060 22872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:05:43.063752 22887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:43.065203 22872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:43.065369 22872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal
uuid: "ba196e9839eb47d881e848d10a0455ac"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-dmhs"
server_key: "308b04ce76df6fc1d257ccc5e39d2a5c"
server_key_iv: "db8e41a0034df6fc27f6a5086cde2788"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.065557 22872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:43.089577 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 3
I20260928 15:05:43.090672 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.090217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54972)
0928 15:05:43.090433 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.090519 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:05:43.091725 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.091390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:54986)
0928 15:05:43.091560 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.091626 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:05:43.091976 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:35957: Network error: Client connection negotiation failed: client connection to 127.9.12.252:35957: connect: Connection refused (error 111)
I20260928 15:05:43.092731 22872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:43.096673 22872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:43.096947 22872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:43.115279 22872 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.252:35957
I20260928 15:05:43.115312 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.252:35957 every 8 connection(s)
I20260928 15:05:43.116662 22872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
I20260928 15:05:43.119473 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 22872
I20260928 15:05:43.119673 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607741729566-9267-0/minicluster-data/master-2/wal/instance
I20260928 15:05:43.120129 9267 external_mini_cluster.cc:1428] Setting key 1aa12ee45cf545ebf87de6efc9b70076
I20260928 15:05:43.120532 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:43.130483 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:43.145525 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.122012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.9.12.252:35957)
0928 15:05:43.122892 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.122907 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.122957 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:05:43.125398 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.125428 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.125957 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.126431 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.127742 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.129760 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.131162 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.132176 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.132399 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.136865 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.136874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.136885 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.136927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.140958 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.141888 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.141898 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.141906 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.142110 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.142558 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.142563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.142566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.143213 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.143499 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.143977 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.144450 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":330,"threads_started":1}
I20260928 15:05:43.149688 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.133299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.9.12.254:42405)
0928 15:05:43.133501 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.133505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.134241 (+ 736us) server_negotiation.cc:407] Connection header received
0928 15:05:43.134437 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.134443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.134514 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.134617 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.135655 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.136422 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.138414 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.139116 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.139237 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.143125 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.143131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.143136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.143177 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.146952 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.147632 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.147637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.147641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.147797 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.148176 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.148182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.148186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.148493 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.148656 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.149109 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.149331 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:05:43.149736 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.133370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42405 (local address 127.0.0.1:34712)
0928 15:05:43.134102 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.134140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.134313 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.134673 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.134680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.135252 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.135497 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.135509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.136568 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.136572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.138250 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.138264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.139312 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.139320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.139508 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.140157 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.140183 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.142927 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.147067 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.147075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.147080 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.147481 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.147905 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.147911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.147914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.148064 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.148620 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.148628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.148812 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.148996 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.149330 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":412,"threads_started":1}
I20260928 15:05:43.151687 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:43.165050 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.152422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:34847 (local address 127.0.0.1:55552)
0928 15:05:43.152582 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.152599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.152685 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.153003 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.153006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.153306 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.153438 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.153445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.154391 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.154397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.155529 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.155544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.156246 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.156252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.156492 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.156964 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.156979 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.159052 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.162291 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.162297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.162303 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.162811 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.163259 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.163265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.163271 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.163454 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.164081 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.164085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.164240 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.164418 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.164680 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:05:43.165323 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.152540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.9.12.253:34847)
0928 15:05:43.152726 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.152731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.152748 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:05:43.152805 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.152811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.152884 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.153000 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.153612 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.154262 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.155674 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.156097 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.156221 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.159223 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.159228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.159232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.159265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.162159 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.162918 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.162923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.162927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.163112 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.163574 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.163578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.163580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.163967 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.164089 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.164565 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.164795 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:05:43.165652 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.150396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.9.12.254:42405)
0928 15:05:43.150581 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.150588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.150609 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:05:43.150765 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.150769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.150822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.150921 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.152421 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.153403 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.154913 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.155316 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.155408 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.159882 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.159886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.159889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.159919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.162566 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.163361 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.163366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.163370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.163531 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.163919 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.163922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.163924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.164246 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.164562 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.164846 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.165072 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:05:43.166261 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:05:43.179049 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.166941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:55014)
0928 15:05:43.167116 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.167136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.167229 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.167582 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.167586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.167943 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.168069 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.168077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.169083 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.169091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.170057 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.170067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.170794 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.170799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.171009 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.171564 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.171593 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.173810 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.176700 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.176707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.176711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.177037 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.177376 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.177382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.177386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.177508 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.177990 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.177994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.178142 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.178274 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.178611 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":5}
I20260928 15:05:43.179068 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.167176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.9.12.252:35957)
0928 15:05:43.167319 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.167326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.167345 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:05:43.167396 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.167401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.167470 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.167562 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.168205 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.168971 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.170204 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.170646 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.170781 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.173977 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.173983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.173987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.174028 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.176583 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.177143 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.177151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.177155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.177261 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.177612 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.177619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.177622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.177876 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.178036 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.178360 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.178763 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":49}
I20260928 15:05:43.181602 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.167356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.9.12.253:34847)
0928 15:05:43.167509 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.167513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.167526 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:05:43.167562 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.167565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.167613 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.167712 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.169013 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.169847 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.171117 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.171697 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.171839 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.176135 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.176140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.176143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.176172 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.178712 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.179464 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.179469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.179473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.179653 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.180086 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.180092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.180095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.180435 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.180771 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.181060 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.181282 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:05:43.184810 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: Bootstrap starting.
I20260928 15:05:43.186223 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:05:43.188102 22939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:43.189229 22939 log.cc:824] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:43.192601 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: No bootstrap required, opened a new log
I20260928 15:05:43.204367 22939 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.205045 22939 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:43.205190 22939 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba196e9839eb47d881e848d10a0455ac, State: Initialized, Role: FOLLOWER
I20260928 15:05:43.207633 22939 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.208065 22952 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:43.208518 22952 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.212713 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.191807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.9.12.254:42405)
0928 15:05:43.191983 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.191989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.192221 (+ 232us) server_negotiation.cc:407] Connection header received
0928 15:05:43.192581 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.192590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.192699 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.192793 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.195802 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.197298 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.199480 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.200252 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.200407 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.204902 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.204908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.204914 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.204956 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.209226 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.210118 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.210127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.210131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.210305 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.210784 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.210793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.210798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.211186 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.211420 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.212056 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.212339 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:05:43.214483 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.191324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.9.12.253:34847)
0928 15:05:43.191506 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.191525 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.192789 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 15:05:43.193055 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.193060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.193145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.193223 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.195413 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.196593 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.199558 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.200225 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.200389 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.204902 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.204908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.204913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.204964 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.211176 (+ 6212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.212105 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.212111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.212114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.212310 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.212753 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.212761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.212765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.213126 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.213340 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.213883 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.214133 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:05:43.217526 22952 leader_election.cc:288] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9573e628338d451e83525719fb21f514 (127.9.12.254:42405), 252fa07124384258ab49f68c77547207 (127.9.12.253:34847)
I20260928 15:05:43.217922 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.218266 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.220484 22939 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:43.221096 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.192990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.9.12.252:35957)
0928 15:05:43.193198 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.193205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.198847 (+ 5642us) server_negotiation.cc:407] Connection header received
0928 15:05:43.199008 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.199015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.199095 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.199199 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.201534 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.202766 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.205015 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.205854 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.205982 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.209517 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.209523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.209530 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.209584 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.216548 (+ 6964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.217380 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.217389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.217396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.218834 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.219253 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.219258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.219261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.219621 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.219920 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.220474 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.220730 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 15:05:43.231801 22964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:05:43.232007 22964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:05:43.227921 22872 master_runner.cc:186] Error getting master registration for 127.9.12.254:42405: OK, instance_id { permanent_uuid: "9573e628338d451e83525719fb21f514" instance_seqno: 1790607942132585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:05:43.239261 22872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:05:43.269184 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 4
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:05:43.285130 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.269873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.9.12.252:35957)
0928 15:05:43.270352 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.270356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.270369 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:05:43.270424 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.270428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.270503 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.270609 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.271558 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.272430 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.274174 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.274888 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.275020 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.279508 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.279514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.279519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.279560 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.282494 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.283285 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.283292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.283294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.283436 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.283806 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.283809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.283810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.284085 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.284189 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.284695 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.284862 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260928 15:05:43.285256 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.269865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:55034)
0928 15:05:43.270068 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.270093 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.270184 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.270643 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.270647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.271208 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.271415 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.271430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.272558 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.272565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.274023 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.274039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.275044 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.275052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.275259 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.275954 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.275982 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.279316 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.282616 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.282622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.282627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.283155 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.283536 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.283541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.283544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.283710 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.284188 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.284194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.284415 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.284585 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.284889 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:05:43.291440 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: Bootstrap starting.
I20260928 15:05:43.296648 22791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:43.297932 22791 log.cc:824] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:43.301218 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: No bootstrap required, opened a new log
I20260928 15:05:43.312191 22791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.312634 22791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:43.312757 22791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9573e628338d451e83525719fb21f514, State: Initialized, Role: FOLLOWER
I20260928 15:05:43.313432 22791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.314435 22967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.314864 22967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.315781 22791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:43.315857 22766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba196e9839eb47d881e848d10a0455ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9573e628338d451e83525719fb21f514" is_pre_election: true
I20260928 15:05:43.316435 22766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba196e9839eb47d881e848d10a0455ac in term 0.
I20260928 15:05:43.317232 22889 leader_election.cc:302] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9573e628338d451e83525719fb21f514, ba196e9839eb47d881e848d10a0455ac; no voters:
I20260928 15:05:43.317902 22953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:43.318114 22953 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:43.318288 22953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:43.323307 22953 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.325359 22953 leader_election.cc:288] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [CANDIDATE]: Term 1 election: Requested vote from peers 9573e628338d451e83525719fb21f514 (127.9.12.254:42405), 252fa07124384258ab49f68c77547207 (127.9.12.253:34847)
I20260928 15:05:43.326068 22766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba196e9839eb47d881e848d10a0455ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9573e628338d451e83525719fb21f514"
I20260928 15:05:43.326443 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:43.329865 22766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba196e9839eb47d881e848d10a0455ac in term 1.
I20260928 15:05:43.330461 22889 leader_election.cc:302] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9573e628338d451e83525719fb21f514, ba196e9839eb47d881e848d10a0455ac; no voters:
I20260928 15:05:43.330816 22953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:43.331017 22953 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [term 1 LEADER]: Becoming Leader. State: Replica: ba196e9839eb47d881e848d10a0455ac, State: Running, Role: LEADER
I20260928 15:05:43.331715 22953 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.336004 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba196e9839eb47d881e848d10a0455ac. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.336472 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:43.338821 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } attempt: 5
I20260928 15:05:43.345984 22974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 15:05:43.349017 22893 tablet.cc:2369] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:43.349339 22974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:05:43 dist-test-slave-dmhs krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
W20260928 15:05:43.372349 22745 tablet.cc:2369] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:43.373158 22766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [term 1 FOLLOWER]: Refusing update from remote peer ba196e9839eb47d881e848d10a0455ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:43.374158 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.339551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:35957 (local address 127.0.0.1:55036)
0928 15:05:43.339898 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.339919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.340034 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.343882 (+ 3848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.343887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.344431 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.344635 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.344647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.346658 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.346665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.348069 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.348084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.349444 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.349450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.349683 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.350501 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.350553 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.354948 (+ 4395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.365691 (+ 10743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.365719 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.365724 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.366538 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.367280 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.367307 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.367311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.367490 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.368226 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.368235 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.368704 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.369014 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.370614 (+ 1600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260928 15:05:43.374239 22952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:43.374804 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.340297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.9.12.252:35957)
0928 15:05:43.340478 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.340484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.340502 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:05:43.340557 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.340563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.340636 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.340739 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.344840 (+ 4101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.346012 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.348268 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.349049 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.349186 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.355262 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.355269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.355274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.355315 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.365465 (+ 10150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.366795 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.366800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.366804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.367067 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.367631 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.367636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.367640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.368000 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.368150 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.369904 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.371287 (+ 1383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:05:43.382774 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: Bootstrap starting.
I20260928 15:05:43.386747 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:43.388238 22865 log.cc:824] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:43.394495 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: No bootstrap required, opened a new log
I20260928 15:05:43.405526 22967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba196e9839eb47d881e848d10a0455ac. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.405920 22967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.406112 22984 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:05:43.412089 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.412670 22865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:43.412823 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252fa07124384258ab49f68c77547207, State: Initialized, Role: FOLLOWER
I20260928 15:05:43.413609 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } }
I20260928 15:05:43.425222 22839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba196e9839eb47d881e848d10a0455ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252fa07124384258ab49f68c77547207"
I20260928 15:05:43.425611 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:43.425520 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.426173 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:43.427703 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.410172 22724 master_runner.cc:186] Error getting master registration for 127.9.12.253:34847: OK, instance_id { permanent_uuid: "252fa07124384258ab49f68c77547207" instance_seqno: 1790607942611661 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:05:43.427980 22724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:05:43.430105 22838 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [term 1 FOLLOWER]: Refusing update from remote peer ba196e9839eb47d881e848d10a0455ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:43.430702 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba196e9839eb47d881e848d10a0455ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252fa07124384258ab49f68c77547207" is_pre_election: true
I20260928 15:05:43.431197 22840 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba196e9839eb47d881e848d10a0455ac in term 1.
I20260928 15:05:43.434031 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.434302 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:43.434607 22953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:43.436364 22974 catalog_manager.cc:1347] Generated new cluster ID: 36f56534fd8c4b38bef1e97baa69f5b5
I20260928 15:05:43.436478 22974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:43.438570 22967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.438822 22967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.446635 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba196e9839eb47d881e848d10a0455ac. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.448077 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.446934 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.448232 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:05:43.452678 23002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:05:43.452802 23002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:05:43.460120 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba196e9839eb47d881e848d10a0455ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9573e628338d451e83525719fb21f514" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42405 } } peers { permanent_uuid: "252fa07124384258ab49f68c77547207" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 34847 } } peers { permanent_uuid: "ba196e9839eb47d881e848d10a0455ac" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 35957 } } }
I20260928 15:05:43.460469 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:43.463599 22974 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:43.465498 22974 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:43.488162 22974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba196e9839eb47d881e848d10a0455ac: Generated new TSK 0
I20260928 15:05:43.490137 22974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:05:44.411255 22984 catalog_manager.cc:1259] Loaded cluster ID: 36f56534fd8c4b38bef1e97baa69f5b5
I20260928 15:05:44.411362 22984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: loading cluster ID for follower catalog manager: success
I20260928 15:05:44.413923 22984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: acquiring CA information for follower catalog manager: success
I20260928 15:05:44.415789 22984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9573e628338d451e83525719fb21f514: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:05:44.454574 23002 catalog_manager.cc:1259] Loaded cluster ID: 36f56534fd8c4b38bef1e97baa69f5b5
I20260928 15:05:44.454674 23002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: loading cluster ID for follower catalog manager: success
I20260928 15:05:44.457067 23002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: acquiring CA information for follower catalog manager: success
I20260928 15:05:44.458776 23002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 252fa07124384258ab49f68c77547207: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:05:44.579211 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22724
I20260928 15:05:44.581105 22786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:44.687006 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22724
I20260928 15:05:44.705142 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22798
I20260928 15:05:44.706250 22860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:44.809912 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22798
I20260928 15:05:44.829330 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 22872
I20260928 15:05:44.830348 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:44.936892 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 22872
2026-09-28T15:05:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4568 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:05:45 dist-test-slave-dmhs krb5kdc[23016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:05:45 dist-test-slave-dmhs krb5kdc[23016](info): set up 2 sockets
Sep 28 15:05:45 dist-test-slave-dmhs krb5kdc[23016](info): commencing operation
krb5kdc: starting...
W20260928 15:05:46.029112 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:05:46 dist-test-slave-dmhs krb5kdc[23016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:05:46.269701 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:41057
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:45645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:41057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:46.592907 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:46.593273 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:46.593370 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:46.600687 23029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:46.600793 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:46.600840 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:46.600888 23029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:46.600925 23029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:46.610726 23029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:45645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:41057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:41057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:46.612012 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:46.613390 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:46.625159 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:46.625228 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:46.625190 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:46.625984 23029 server_base.cc:1034] running on GCE node
I20260928 15:05:46.626722 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:46.628064 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:46.629261 23029 hybrid_clock.cc:648] HybridClock initialized: now 1790607946629201 us; error 48 us; skew 500 ppm
I20260928 15:05:46.629624 23029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:46 dist-test-slave-dmhs krb5kdc[23016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:46.632962 23029 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:46.634473 23029 webserver.cc:460] Webserver started at http://127.9.12.254:39241/ using document root <none> and password file <none>
I20260928 15:05:46.635149 23029 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:46.635290 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:46.635596 23029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:46.638584 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "0893a5015ce146e8863271ef45ee7407"
format_stamp: "Formatted at 2026-09-28 15:05:46 on dist-test-slave-dmhs"
server_key: "b64921895ddf08176db7d35f1b4a2bb4"
server_key_iv: "3535980bdc21caa7538c3de0f4f1865c"
server_key_version: "encryptionkey@0"
I20260928 15:05:46.639446 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "0893a5015ce146e8863271ef45ee7407"
format_stamp: "Formatted at 2026-09-28 15:05:46 on dist-test-slave-dmhs"
server_key: "b64921895ddf08176db7d35f1b4a2bb4"
server_key_iv: "3535980bdc21caa7538c3de0f4f1865c"
server_key_version: "encryptionkey@0"
I20260928 15:05:46.644657 23029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:05:46.648002 23044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:46.649365 23029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:46.649549 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "0893a5015ce146e8863271ef45ee7407"
format_stamp: "Formatted at 2026-09-28 15:05:46 on dist-test-slave-dmhs"
server_key: "b64921895ddf08176db7d35f1b4a2bb4"
server_key_iv: "3535980bdc21caa7538c3de0f4f1865c"
server_key_version: "encryptionkey@0"
I20260928 15:05:46.649757 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:46.665431 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:46.669763 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:46.670053 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:46.688062 23029 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:41057
I20260928 15:05:46.688097 23095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:41057 every 8 connection(s)
I20260928 15:05:46.689502 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:46.691015 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23029
I20260928 15:05:46.691236 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:05:46.692076 9267 external_mini_cluster.cc:1428] Setting key 9c630ba377f5223d479df9753160019e
I20260928 15:05:46.694314 23096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:05:46 dist-test-slave-dmhs krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:46.707803 23096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407: Bootstrap starting.
I20260928 15:05:46.712013 23096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:46.713315 23096 log.cc:824] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:46.716666 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:46.694090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.9.12.254:41057)
0928 15:05:46.694882 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:46.694893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:46.694933 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:05:46.696597 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:46.696617 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:46.696979 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:46.697293 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:46.699158 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:46.700536 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:46.702382 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:46.703029 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:46.703147 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:46.707868 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:46.707880 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:46.707902 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:46.707954 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:46.711882 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:46.712745 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:46.712753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:46.712760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:46.712975 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:46.713347 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:46.713352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:46.713354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:46.714064 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:46.714327 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:46.714898 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 15:05:46.715305 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":377,"threads_started":1}
I20260928 15:05:46.716715 23096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407: No bootstrap required, opened a new log
I20260928 15:05:46.720444 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:05:46.724941 23096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0893a5015ce146e8863271ef45ee7407" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41057 } }
I20260928 15:05:46.725324 23096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:46.725442 23096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0893a5015ce146e8863271ef45ee7407, State: Initialized, Role: FOLLOWER
I20260928 15:05:46.726060 23096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0893a5015ce146e8863271ef45ee7407" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41057 } }
I20260928 15:05:46.726279 23096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:46.726397 23096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:46.726560 23096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:46.728897 23096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0893a5015ce146e8863271ef45ee7407" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41057 } }
I20260928 15:05:46.729393 23096 leader_election.cc:302] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0893a5015ce146e8863271ef45ee7407; no voters:
I20260928 15:05:46.730005 23096 leader_election.cc:288] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:05:46.730150 23101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:46.730822 23101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [term 1 LEADER]: Becoming Leader. State: Replica: 0893a5015ce146e8863271ef45ee7407, State: Running, Role: LEADER
I20260928 15:05:46.731323 23101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0893a5015ce146e8863271ef45ee7407" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41057 } }
I20260928 15:05:46.732064 23096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:46.734110 23102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0893a5015ce146e8863271ef45ee7407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0893a5015ce146e8863271ef45ee7407" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41057 } } }
I20260928 15:05:46.734542 23102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:46.735394 23103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0893a5015ce146e8863271ef45ee7407. Latest consensus state: current_term: 1 leader_uuid: "0893a5015ce146e8863271ef45ee7407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0893a5015ce146e8863271ef45ee7407" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 41057 } } }
I20260928 15:05:46.735723 23103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:46.735885 23109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:46.741106 23109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:46.749735 23109 catalog_manager.cc:1347] Generated new cluster ID: 160de6f1c5244becbeb187df2be146c7
I20260928 15:05:46.749836 23109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:46.784708 23109 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:46.786049 23109 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:46.803665 23109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0893a5015ce146e8863271ef45ee7407: Generated new TSK 0
I20260928 15:05:46.804733 23109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:05:46.832567 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23029
I20260928 15:05:46.833877 23091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:46.930866 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23029
2026-09-28T15:05:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1991 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:05:47 dist-test-slave-dmhs krb5kdc[23124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:05:47 dist-test-slave-dmhs krb5kdc[23124](info): set up 2 sockets
Sep 28 15:05:47 dist-test-slave-dmhs krb5kdc[23124](info): commencing operation
krb5kdc: starting...
W20260928 15:05:48.025058 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:05:48 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:05:48.275825 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33729
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:48.604036 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:48.604382 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:48.604499 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:48.611824 23137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:48.611924 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:48.611980 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:48.612037 23137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:48.612074 23137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:48.622118 23137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:33729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:48.623371 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:48.624663 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:48.636991 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.636902 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.636935 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:48.638108 23137 server_base.cc:1034] running on GCE node
I20260928 15:05:48.638857 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:48.640300 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:48.641525 23137 hybrid_clock.cc:648] HybridClock initialized: now 1790607948641464 us; error 55 us; skew 500 ppm
I20260928 15:05:48.641896 23137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:48 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:48.649288 23137 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:48.650857 23137 webserver.cc:460] Webserver started at http://127.9.12.254:43995/ using document root <none> and password file <none>
I20260928 15:05:48.651573 23137 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:48.651715 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:48.652016 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:48.654973 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "8f2f821d90a040139832b7d16c06a3a6"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-dmhs"
server_key: "3da9675cc9806d94b1693e88da42fcdb"
server_key_iv: "34a9f116ab0112bc4a2959555be06f08"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.655716 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "8f2f821d90a040139832b7d16c06a3a6"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-dmhs"
server_key: "3da9675cc9806d94b1693e88da42fcdb"
server_key_iv: "34a9f116ab0112bc4a2959555be06f08"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.661008 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:05:48.664471 23152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.665990 23137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:05:48.666188 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "8f2f821d90a040139832b7d16c06a3a6"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-dmhs"
server_key: "3da9675cc9806d94b1693e88da42fcdb"
server_key_iv: "34a9f116ab0112bc4a2959555be06f08"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.666405 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:48.689468 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:48.693517 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:48.693812 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:48.712543 23137 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:33729
I20260928 15:05:48.712563 23203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:33729 every 8 connection(s)
I20260928 15:05:48.713904 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:48.717476 23204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:48.718954 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23137
I20260928 15:05:48.719164 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:05:48.719812 9267 external_mini_cluster.cc:1428] Setting key 17834d76e3aa47be9b4314a2f068d6f1
I20260928 15:05:48.730435 23204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap starting.
I20260928 15:05:48.734057 23204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Neither blocks nor log segments found. Creating new log.
Sep 28 15:05:48 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:48.735085 23204 log.cc:824] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.738662 23204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: No bootstrap required, opened a new log
I20260928 15:05:48.744696 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.721934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.9.12.254:33729)
0928 15:05:48.722722 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.722737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.722790 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:05:48.725164 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.725194 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.725674 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.726098 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.727387 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.729459 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.730788 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.731767 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.731957 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.736173 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.736181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.736194 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.736230 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.740173 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.740918 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.740931 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.740940 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.741164 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.741520 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.741525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.741528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.742248 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.742548 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.742869 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.743325 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":317,"threads_started":1}
I20260928 15:05:48.748183 23204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:48.748554 23204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:48.748631 23204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2f821d90a040139832b7d16c06a3a6, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.749118 23204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:48.749307 23204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:48.749401 23204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:48.749531 23204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:48.751549 23204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:48.752020 23204 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2f821d90a040139832b7d16c06a3a6; no voters:
I20260928 15:05:48.752652 23204 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:05:48.752816 23209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:48.754323 23209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8f2f821d90a040139832b7d16c06a3a6, State: Running, Role: LEADER
I20260928 15:05:48.754833 23204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:48.754844 23209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:48.757710 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2f821d90a040139832b7d16c06a3a6. Latest consensus state: current_term: 1 leader_uuid: "8f2f821d90a040139832b7d16c06a3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } } }
I20260928 15:05:48.758092 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:48.758077 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f2f821d90a040139832b7d16c06a3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } } }
I20260928 15:05:48.758412 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:48.761453 23218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:48.766994 23218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:48.776611 23218 catalog_manager.cc:1347] Generated new cluster ID: f9421391b33e476297b8a7ad83c16b5e
I20260928 15:05:48.776743 23218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:48.790544 23218 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:48.791846 23218 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:48.804445 23218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Generated new TSK 0
I20260928 15:05:48.805316 23218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:05:48.859268 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33729
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:49.173436 23231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.173707 23231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.173887 23231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.181038 23231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.181182 23231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.181351 23231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:05:49.190691 23231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:49.191848 23231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.193130 23231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:49.206101 23236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.206187 23237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.206161 23239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.207006 23231 server_base.cc:1034] running on GCE node
I20260928 15:05:49.207584 23231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.208585 23231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.209807 23231 hybrid_clock.cc:648] HybridClock initialized: now 1790607949209768 us; error 50 us; skew 500 ppm
I20260928 15:05:49.210117 23231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:49 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.213100 23231 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:05:49.214545 23231 webserver.cc:460] Webserver started at http://127.9.12.193:46275/ using document root <none> and password file <none>
I20260928 15:05:49.215262 23231 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.215399 23231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.215698 23231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:49.218585 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "6c9d5b53831942c0b539f604604fb793"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "b8a51776f49dd29b86400ac81496ed54"
server_key_iv: "1a898eded9e00ee86896956c9f0dc01e"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.219326 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "6c9d5b53831942c0b539f604604fb793"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "b8a51776f49dd29b86400ac81496ed54"
server_key_iv: "1a898eded9e00ee86896956c9f0dc01e"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.224141 23231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 15:05:49.227372 23246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.228670 23231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:49.228863 23231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "6c9d5b53831942c0b539f604604fb793"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "b8a51776f49dd29b86400ac81496ed54"
server_key_iv: "1a898eded9e00ee86896956c9f0dc01e"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.229053 23231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.252842 23231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.256794 23231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.257091 23231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.257984 23231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:49.259366 23231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:49.259465 23231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.259625 23231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:49.259694 23231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.290552 23231 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:46463
I20260928 15:05:49.290601 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:46463 every 8 connection(s)
I20260928 15:05:49.291877 23231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:49.301694 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23231
I20260928 15:05:49.301909 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:05:49.302544 9267 external_mini_cluster.cc:1428] Setting key 928f3d5cdeb7f8b1ac6a20e23ebcc77e
Sep 28 15:05:49 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:49.311878 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.294414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:37527 (local address 127.9.12.254:33729)
0928 15:05:49.294855 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.294860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.295719 (+ 859us) server_negotiation.cc:407] Connection header received
0928 15:05:49.297568 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.297573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.297660 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.297775 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:49.299224 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.299957 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.301458 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.301918 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.302049 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.306146 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:49.306151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:49.306154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:49.306184 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:49.308966 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.309701 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.309706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.309708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.309826 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.310262 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.310267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.310271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.310535 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:49.310692 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.311443 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.311620 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":239,"threads_started":1}
I20260928 15:05:49.313373 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.294893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:37527)
0928 15:05:49.295571 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.295606 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.297298 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.298004 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:49.298012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:49.298488 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:49.299043 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.299068 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.300139 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.300143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.301323 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.301335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.302263 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.302267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.302459 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.303279 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:49.303311 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:49.305928 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:49.309116 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:49.309125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:49.309141 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:49.309568 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:49.309950 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:49.309954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:49.309958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:49.310145 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:49.310668 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:49.310676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:49.311034 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 15:05:49.311353 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 15:05:49.311944 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260928 15:05:49.314925 23359 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33729
I20260928 15:05:49.315289 23359 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:49.315979 23359 heartbeater.cc:511] Master 127.9.12.254:33729 requested a full tablet report, sending...
I20260928 15:05:49.317960 23169 ts_manager.cc:194] Registered new tserver with Master: 6c9d5b53831942c0b539f604604fb793 (127.9.12.193:46463)
I20260928 15:05:49.319337 23169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:37527
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:05:49.356137 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33729
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:49.700193 23365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.700500 23365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.700731 23365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.707813 23365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.707955 23365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.708150 23365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:05:49.717690 23365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:49.718904 23365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.720158 23365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:49.732878 23371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.732878 23373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.733582 23370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.734910 23365 server_base.cc:1034] running on GCE node
I20260928 15:05:49.735440 23365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.736326 23365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.737558 23365 hybrid_clock.cc:648] HybridClock initialized: now 1790607949737479 us; error 68 us; skew 500 ppm
I20260928 15:05:49.737836 23365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:49 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.740882 23365 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:05:49.742370 23365 webserver.cc:460] Webserver started at http://127.9.12.194:32813/ using document root <none> and password file <none>
I20260928 15:05:49.743027 23365 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.743121 23365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.743351 23365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:49.746280 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "6e8a3518315c4797907b988921c959de"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "f14655277445d293f91f54a79a1da08c"
server_key_iv: "8ddd12e909d3c3d6de55d20dcf9768b5"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.746989 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "6e8a3518315c4797907b988921c959de"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "f14655277445d293f91f54a79a1da08c"
server_key_iv: "8ddd12e909d3c3d6de55d20dcf9768b5"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.751796 23365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 15:05:49.755008 23380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.756345 23365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:49.756469 23365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "6e8a3518315c4797907b988921c959de"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "f14655277445d293f91f54a79a1da08c"
server_key_iv: "8ddd12e909d3c3d6de55d20dcf9768b5"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.756636 23365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.771274 23365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.775149 23365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.775378 23365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.776235 23365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:49.777524 23365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:49.777594 23365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.777684 23365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:49.777729 23365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.813002 23365 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:41555
I20260928 15:05:49.813050 23492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:41555 every 8 connection(s)
I20260928 15:05:49.814433 23365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:49.823110 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23365
I20260928 15:05:49.823283 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:05:49.823724 9267 external_mini_cluster.cc:1428] Setting key db6c7f0d5e6ff8b9d3357e8db0378aa6
Sep 28 15:05:49 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:49.836108 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.817100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:38085 (local address 127.9.12.254:33729)
0928 15:05:49.817610 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.817614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.818393 (+ 779us) server_negotiation.cc:407] Connection header received
0928 15:05:49.820472 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.820478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.820564 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.820683 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:49.822119 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.822904 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.824713 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.825167 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.825260 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.829111 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:49.829120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:49.829128 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:49.829175 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:49.832752 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.833630 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.833638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.833644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.833808 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.834249 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.834255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.834261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.834638 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:49.834758 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.835667 (+ 909us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.835836 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":286,"threads_started":1}
I20260928 15:05:49.837603 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.817535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:38085)
0928 15:05:49.818229 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.818273 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.820153 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.820918 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:49.820925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:49.821421 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:49.821939 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.821954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.823059 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.823063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.824592 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.824604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.825395 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.825405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.825762 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.826457 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:49.826478 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:49.828861 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:49.832933 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:49.832945 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:49.832957 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:49.833493 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:49.833931 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:49.833934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:49.833936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:49.834135 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:49.834764 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:49.834775 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:49.835252 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 15:05:49.835573 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 15:05:49.836166 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":271,"threads_started":1}
I20260928 15:05:49.839166 23493 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33729
I20260928 15:05:49.839479 23493 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:49.840368 23493 heartbeater.cc:511] Master 127.9.12.254:33729 requested a full tablet report, sending...
I20260928 15:05:49.842018 23169 ts_manager.cc:194] Registered new tserver with Master: 6e8a3518315c4797907b988921c959de (127.9.12.194:41555)
I20260928 15:05:49.843252 23169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:38085
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:05:49.882995 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33729
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:50.205675 23499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:50.206003 23499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:50.206172 23499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:50.213444 23499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:50.213598 23499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:50.213771 23499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:05:50.223212 23499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:50.224591 23499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:50.225900 23499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:50.238811 23504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:50.238888 23505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:50.238910 23507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:50.239702 23499 server_base.cc:1034] running on GCE node
I20260928 15:05:50.240310 23499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:50.241216 23499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:50.242427 23499 hybrid_clock.cc:648] HybridClock initialized: now 1790607950242354 us; error 53 us; skew 500 ppm
I20260928 15:05:50.242740 23499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:50 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:50.245949 23499 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:05:50.247359 23499 webserver.cc:460] Webserver started at http://127.9.12.195:36363/ using document root <none> and password file <none>
I20260928 15:05:50.248018 23499 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:50.248126 23499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:50.248464 23499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:50.251317 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "82b346941c2949d88bbd345c40d5a91f"
format_stamp: "Formatted at 2026-09-28 15:05:50 on dist-test-slave-dmhs"
server_key: "77ba0fe67c5df692cc8f2df151e86f86"
server_key_iv: "c59036e62196523ce8e781590100eafc"
server_key_version: "encryptionkey@0"
I20260928 15:05:50.252079 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "82b346941c2949d88bbd345c40d5a91f"
format_stamp: "Formatted at 2026-09-28 15:05:50 on dist-test-slave-dmhs"
server_key: "77ba0fe67c5df692cc8f2df151e86f86"
server_key_iv: "c59036e62196523ce8e781590100eafc"
server_key_version: "encryptionkey@0"
I20260928 15:05:50.257519 23499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:05:50.261322 23514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:50.262660 23499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:05:50.262826 23499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "82b346941c2949d88bbd345c40d5a91f"
format_stamp: "Formatted at 2026-09-28 15:05:50 on dist-test-slave-dmhs"
server_key: "77ba0fe67c5df692cc8f2df151e86f86"
server_key_iv: "c59036e62196523ce8e781590100eafc"
server_key_version: "encryptionkey@0"
I20260928 15:05:50.263063 23499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:50.282207 23499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:50.286006 23499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:50.286284 23499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:50.287231 23499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:50.288517 23499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:50.288587 23499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:50.288769 23499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:50.288820 23499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:50.321543 23499 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:42315
I20260928 15:05:50.321589 23626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:42315 every 8 connection(s)
I20260928 15:05:50.322196 23359 heartbeater.cc:503] Master 127.9.12.254:33729 was elected leader, sending a full tablet report...
I20260928 15:05:50.323032 23499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:50.325544 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23499
I20260928 15:05:50.325747 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:05:50.326273 9267 external_mini_cluster.cc:1428] Setting key 5d9025cc5677dcb8e6a507db7bc245ac
Sep 28 15:05:50 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:50.343323 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.325701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:34275 (local address 127.9.12.254:33729)
0928 15:05:50.325858 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.325864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.327403 (+ 1539us) server_negotiation.cc:407] Connection header received
0928 15:05:50.329694 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.329698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.329771 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.329850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:50.331160 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.331851 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.333513 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.333969 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.334061 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.337299 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:50.337303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:50.337306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:50.337337 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:50.340195 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.341045 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.341052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.341055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.341208 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.341677 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.341681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.341684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.341962 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:50.342136 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.342766 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.342995 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:05:50.344378 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.326319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:34275)
0928 15:05:50.327222 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.327274 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:50.329471 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:50.330054 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:50.330062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:50.330526 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:50.331031 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.331046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.331974 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.331978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.333385 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.333398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.334177 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.334181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.334389 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.334973 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:50.334998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:50.337116 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:50.340350 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:50.340358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:50.340386 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:50.340940 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:50.341315 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:50.341319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:50.341322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:50.341549 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:50.342100 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:50.342107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:50.342429 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 15:05:50.342694 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 15:05:50.343209 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":314,"threads_started":1}
I20260928 15:05:50.345721 23627 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33729
I20260928 15:05:50.346026 23627 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:50.346724 23627 heartbeater.cc:511] Master 127.9.12.254:33729 requested a full tablet report, sending...
I20260928 15:05:50.348184 23169 ts_manager.cc:194] Registered new tserver with Master: 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315)
I20260928 15:05:50.349025 23169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:34275
I20260928 15:05:50.353752 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:50.371479 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.358302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.9.12.254:33729)
0928 15:05:50.358465 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.358470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.358994 (+ 524us) server_negotiation.cc:407] Connection header received
0928 15:05:50.359200 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.359204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.359262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.359379 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:50.360925 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.361684 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.363044 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.363584 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.363723 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.366186 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:50.366192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:50.366197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:50.366238 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:50.368726 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.369383 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.369390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.369393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.369532 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.369922 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.369930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.369941 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.370261 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:50.370542 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.370903 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.371139 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:05:50.375356 23169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:05:50.382112 23169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:05:50.403855 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.393158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:42315 (local address 127.0.0.1:32916)
0928 15:05:50.394088 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.394150 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:50.394319 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:50.396425 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:50.396434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:50.396485 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:50.397052 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.397068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.399138 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.399144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.400738 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.400753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.402878 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.402887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.403105 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.403166 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 15:05:50.403305 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 15:05:50.403504 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":534,"threads_started":1}
I20260928 15:05:50.403882 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.393428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.9.12.195:42315)
0928 15:05:50.395669 (+ 2241us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.395680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.395718 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:05:50.395832 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.395844 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.396129 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.396362 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:50.397469 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.399013 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.400951 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.402406 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.402564 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.402797 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.403373 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.403552 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2038,"thread_start_us":1878,"threads_started":1}
I20260928 15:05:50.405754 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.393485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:46463 (local address 127.0.0.1:47932)
0928 15:05:50.394880 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.394902 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:50.395046 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:50.398033 (+ 2987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:50.398037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:50.398066 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:50.398542 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.398553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.400684 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.400687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.402807 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.402821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.404692 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.404698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.404864 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.404896 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:05:50.405221 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 15:05:50.405356 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":1,"thread_start_us":1041,"threads_started":1}
I20260928 15:05:50.405773 23642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.394168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.9.12.193:46463)
0928 15:05:50.397440 (+ 3272us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.397449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.397475 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:05:50.397569 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.397578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.397798 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.398756 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:50.398997 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.400506 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.402981 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.404480 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.404618 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.404832 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.405282 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.405459 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3090,"thread_start_us":2947,"threads_started":1}
I20260928 15:05:50.407092 23643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.394863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:41555 (local address 127.0.0.1:49074)
0928 15:05:50.397125 (+ 2262us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.397144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:50.397265 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:50.398159 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:50.398167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:50.398192 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:50.398628 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.398641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.400589 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.400595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.402978 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.402992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.406329 (+ 3337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.406335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.406505 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.406536 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:05:50.406605 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:05:50.406720 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"mutex_wait_us":3,"thread_start_us":1959,"threads_started":1}
I20260928 15:05:50.407599 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.395309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.9.12.194:41555)
0928 15:05:50.395881 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.395893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.397367 (+ 1474us) server_negotiation.cc:407] Connection header received
0928 15:05:50.397488 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.397495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.397734 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.397939 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:50.398812 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.400441 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.403138 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.404913 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.406803 (+ 1890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.407008 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.407084 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.407250 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":304,"threads_started":1}
I20260928 15:05:50.408144 23562 tablet_service.cc:1467] Processing CreateTablet for tablet de17e89fb63f4cc485d709770f525c60 (DEFAULT_TABLE table=test-table [id=77d2b61499a147cbb8b8a39e529f4e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:50.409191 23294 tablet_service.cc:1467] Processing CreateTablet for tablet de17e89fb63f4cc485d709770f525c60 (DEFAULT_TABLE table=test-table [id=77d2b61499a147cbb8b8a39e529f4e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:50.409734 23562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de17e89fb63f4cc485d709770f525c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:50.410917 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de17e89fb63f4cc485d709770f525c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:50.410880 23428 tablet_service.cc:1467] Processing CreateTablet for tablet de17e89fb63f4cc485d709770f525c60 (DEFAULT_TABLE table=test-table [id=77d2b61499a147cbb8b8a39e529f4e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:50.412125 23428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de17e89fb63f4cc485d709770f525c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:50.421722 23645 tablet_bootstrap.cc:492] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Bootstrap starting.
I20260928 15:05:50.424685 23646 tablet_bootstrap.cc:492] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Bootstrap starting.
I20260928 15:05:50.426263 23645 tablet_bootstrap.cc:654] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:50.426303 23647 tablet_bootstrap.cc:492] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Bootstrap starting.
I20260928 15:05:50.428107 23645 log.cc:824] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:50.428218 23646 tablet_bootstrap.cc:654] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:50.429282 23647 tablet_bootstrap.cc:654] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:50.429409 23646 log.cc:824] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:50.430431 23647 log.cc:824] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:50.431358 23645 tablet_bootstrap.cc:492] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: No bootstrap required, opened a new log
I20260928 15:05:50.431648 23645 ts_tablet_manager.cc:1397] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 15:05:50.433382 23647 tablet_bootstrap.cc:492] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: No bootstrap required, opened a new log
I20260928 15:05:50.433665 23647 ts_tablet_manager.cc:1397] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:05:50.437234 23646 tablet_bootstrap.cc:492] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: No bootstrap required, opened a new log
I20260928 15:05:50.437508 23646 ts_tablet_manager.cc:1397] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20260928 15:05:50.439394 23645 raft_consensus.cc:348] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.439756 23645 raft_consensus.cc:374] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:50.439882 23645 raft_consensus.cc:729] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9d5b53831942c0b539f604604fb793, State: Initialized, Role: FOLLOWER
I20260928 15:05:50.440572 23645 consensus_queue.cc:260] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.442233 23645 ts_tablet_manager.cc:1428] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260928 15:05:50.442982 23647 raft_consensus.cc:348] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.443375 23647 raft_consensus.cc:374] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:50.443463 23647 raft_consensus.cc:729] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e8a3518315c4797907b988921c959de, State: Initialized, Role: FOLLOWER
I20260928 15:05:50.444083 23647 consensus_queue.cc:260] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.445276 23493 heartbeater.cc:503] Master 127.9.12.254:33729 was elected leader, sending a full tablet report...
I20260928 15:05:50.445770 23647 ts_tablet_manager.cc:1428] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 15:05:50.448127 23646 raft_consensus.cc:348] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.448562 23646 raft_consensus.cc:374] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:50.448660 23646 raft_consensus.cc:729] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b346941c2949d88bbd345c40d5a91f, State: Initialized, Role: FOLLOWER
I20260928 15:05:50.449313 23646 consensus_queue.cc:260] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.450505 23627 heartbeater.cc:503] Master 127.9.12.254:33729 was elected leader, sending a full tablet report...
I20260928 15:05:50.451242 23646 ts_tablet_manager.cc:1428] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260928 15:05:50.543092 23651 raft_consensus.cc:482] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:50.543437 23651 raft_consensus.cc:504] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
W20260928 15:05:50.544286 23360 tablet.cc:2369] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:50.545567 23651 leader_election.cc:288] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315), 6e8a3518315c4797907b988921c959de (127.9.12.194:41555)
I20260928 15:05:50.554432 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.546050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:48303 (local address 127.9.12.195:42315)
0928 15:05:50.546530 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.546539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.547060 (+ 521us) server_negotiation.cc:407] Connection header received
0928 15:05:50.547335 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.547340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.547414 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.547500 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:50.548318 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.549054 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.551978 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.553274 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.553443 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.553506 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.554033 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.554178 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260928 15:05:50.554433 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.546029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:34317 (local address 127.9.12.194:41555)
0928 15:05:50.546320 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.546330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.546791 (+ 461us) server_negotiation.cc:407] Connection header received
0928 15:05:50.547111 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.547119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.547221 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.547344 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:50.548075 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.548906 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.551944 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.553273 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.553429 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.553502 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.554039 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.554178 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 15:05:50.554697 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.546012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:42315 (local address 127.9.12.193:48303)
0928 15:05:50.546967 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.546983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:50.547173 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:50.547562 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:50.547566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:50.547591 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:50.548040 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.548052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.549187 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.549192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.551743 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.551762 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.553554 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.553562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.553852 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.553897 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 15:05:50.553991 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:05:50.554225 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":250,"thread_start_us":657,"threads_started":1}
I20260928 15:05:50.554751 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.545820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:41555 (local address 127.9.12.193:34317)
0928 15:05:50.546649 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.546696 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:50.546947 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:50.547373 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:50.547377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:50.547403 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:50.547869 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.547885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.549054 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.549058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.551760 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:50.551769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.553572 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.553577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:50.553838 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.553890 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 15:05:50.554002 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 15:05:50.554225 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":344,"thread_start_us":475,"threads_started":1}
I20260928 15:05:50.555759 23448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de17e89fb63f4cc485d709770f525c60" candidate_uuid: "6c9d5b53831942c0b539f604604fb793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e8a3518315c4797907b988921c959de" is_pre_election: true
I20260928 15:05:50.555759 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de17e89fb63f4cc485d709770f525c60" candidate_uuid: "6c9d5b53831942c0b539f604604fb793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b346941c2949d88bbd345c40d5a91f" is_pre_election: true
I20260928 15:05:50.556345 23582 raft_consensus.cc:2454] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9d5b53831942c0b539f604604fb793 in term 0.
I20260928 15:05:50.556345 23448 raft_consensus.cc:2454] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9d5b53831942c0b539f604604fb793 in term 0.
I20260928 15:05:50.556982 23250 leader_election.cc:302] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9d5b53831942c0b539f604604fb793, 6e8a3518315c4797907b988921c959de; no voters:
I20260928 15:05:50.557443 23651 raft_consensus.cc:2781] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:50.557590 23651 raft_consensus.cc:482] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:50.557701 23651 raft_consensus.cc:3037] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:50.560058 23651 raft_consensus.cc:504] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.560791 23651 leader_election.cc:288] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [CANDIDATE]: Term 1 election: Requested vote from peers 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315), 6e8a3518315c4797907b988921c959de (127.9.12.194:41555)
I20260928 15:05:50.561182 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de17e89fb63f4cc485d709770f525c60" candidate_uuid: "6c9d5b53831942c0b539f604604fb793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b346941c2949d88bbd345c40d5a91f"
I20260928 15:05:50.561260 23448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de17e89fb63f4cc485d709770f525c60" candidate_uuid: "6c9d5b53831942c0b539f604604fb793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e8a3518315c4797907b988921c959de"
I20260928 15:05:50.561408 23582 raft_consensus.cc:3037] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:50.561457 23448 raft_consensus.cc:3037] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:50.564699 23448 raft_consensus.cc:2454] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9d5b53831942c0b539f604604fb793 in term 1.
I20260928 15:05:50.564699 23582 raft_consensus.cc:2454] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9d5b53831942c0b539f604604fb793 in term 1.
I20260928 15:05:50.565090 23250 leader_election.cc:302] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9d5b53831942c0b539f604604fb793, 6e8a3518315c4797907b988921c959de; no voters:
I20260928 15:05:50.565430 23651 raft_consensus.cc:2781] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:50.565876 23651 raft_consensus.cc:686] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 1 LEADER]: Becoming Leader. State: Replica: 6c9d5b53831942c0b539f604604fb793, State: Running, Role: LEADER
W20260928 15:05:50.566560 23494 tablet.cc:2369] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:50.566414 23651 consensus_queue.cc:237] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } }
I20260928 15:05:50.571107 23168 catalog_manager.cc:5495] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c9d5b53831942c0b539f604604fb793 (127.9.12.193). New cstate: current_term: 1 leader_uuid: "6c9d5b53831942c0b539f604604fb793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } health_report { overall_health: UNKNOWN } } }
W20260928 15:05:50.574828 23628 tablet.cc:2369] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:50.595940 23642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.591206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.9.12.193:46463)
0928 15:05:50.591365 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.591369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.591386 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:05:50.591598 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.591601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.591675 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.591792 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:50.592097 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.592828 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.594173 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.594586 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.594666 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.595067 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:50.595528 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:50.595632 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.595698 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.595758 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:05:50.603540 23582 raft_consensus.cc:1261] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Refusing update from remote peer 6c9d5b53831942c0b539f604604fb793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:50.603667 23448 raft_consensus.cc:1261] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Refusing update from remote peer 6c9d5b53831942c0b539f604604fb793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:50.604611 23656 consensus_queue.cc:1035] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:50.604933 23651 consensus_queue.cc:1035] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:50.616672 23659 mvcc.cc:205] Tried to move back new op lower bound from 7334330165662339072 to 7334330165524619264. Current Snapshot: MvccSnapshot[applied={T|T < 7334330165662339072}]
I20260928 15:05:50.630606 23168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:50.630865 23168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:50.634547 23168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Sending DeleteTablet for 3 replicas of tablet de17e89fb63f4cc485d709770f525c60
I20260928 15:05:50.635735 23562 tablet_service.cc:1514] Processing DeleteTablet for tablet de17e89fb63f4cc485d709770f525c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:50 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:32916
I20260928 15:05:50.635826 23428 tablet_service.cc:1514] Processing DeleteTablet for tablet de17e89fb63f4cc485d709770f525c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:50 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:49074
I20260928 15:05:50.636117 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23231
I20260928 15:05:50.636791 23294 tablet_service.cc:1514] Processing DeleteTablet for tablet de17e89fb63f4cc485d709770f525c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:50 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:47932
I20260928 15:05:50.650602 23670 tablet_replica.cc:330] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: stopping tablet replica
I20260928 15:05:50.652839 23670 raft_consensus.cc:2229] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:50.653420 23672 tablet_replica.cc:330] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: stopping tablet replica
I20260928 15:05:50.653640 23670 raft_consensus.cc:2258] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:50.653831 23672 raft_consensus.cc:2229] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:05:50.656087 23672 raft_consensus.cc:2258] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:50.656678 23670 ts_tablet_manager.cc:1905] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:50.656708 23669 tablet_replica.cc:330] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: stopping tablet replica
I20260928 15:05:50.657078 23669 raft_consensus.cc:2229] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:50.657562 23669 raft_consensus.cc:2258] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:50.658361 23672 ts_tablet_manager.cc:1905] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:50.659366 23669 ts_tablet_manager.cc:1905] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:05:50 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:05:50.664547 23670 ts_tablet_manager.cc:1918] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:50.664776 23670 log.cc:1196] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/de17e89fb63f4cc485d709770f525c60
I20260928 15:05:50.665406 23670 ts_tablet_manager.cc:1939] T de17e89fb63f4cc485d709770f525c60 P 6e8a3518315c4797907b988921c959de: Deleting consensus metadata
I20260928 15:05:50.666210 23669 ts_tablet_manager.cc:1918] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:50.666352 23669 log.cc:1196] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/de17e89fb63f4cc485d709770f525c60
I20260928 15:05:50.666925 23669 ts_tablet_manager.cc:1939] T de17e89fb63f4cc485d709770f525c60 P 82b346941c2949d88bbd345c40d5a91f: Deleting consensus metadata
I20260928 15:05:50.667630 23156 catalog_manager.cc:4841] TS 6e8a3518315c4797907b988921c959de (127.9.12.194:41555): tablet de17e89fb63f4cc485d709770f525c60 (table test-table [id=77d2b61499a147cbb8b8a39e529f4e23]) successfully deleted
I20260928 15:05:50.668756 23154 catalog_manager.cc:4841] TS 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315): tablet de17e89fb63f4cc485d709770f525c60 (table test-table [id=77d2b61499a147cbb8b8a39e529f4e23]) successfully deleted
I20260928 15:05:50.669736 23672 ts_tablet_manager.cc:1918] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:50.669919 23672 log.cc:1196] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/de17e89fb63f4cc485d709770f525c60
I20260928 15:05:50.670445 23672 ts_tablet_manager.cc:1939] T de17e89fb63f4cc485d709770f525c60 P 6c9d5b53831942c0b539f604604fb793: Deleting consensus metadata
I20260928 15:05:50.671885 23156 catalog_manager.cc:4841] TS 6c9d5b53831942c0b539f604604fb793 (127.9.12.193:46463): tablet de17e89fb63f4cc485d709770f525c60 (table test-table [id=77d2b61499a147cbb8b8a39e529f4e23]) successfully deleted
I20260928 15:05:50.672684 23642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.637322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.9.12.193:46463)
0928 15:05:50.637509 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.637515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.648525 (+ 11010us) server_negotiation.cc:407] Connection header received
0928 15:05:50.649086 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.649095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.649206 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.649287 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:50.654492 (+ 5205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.655915 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.658034 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.658688 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.658819 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.664775 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:50.664789 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:50.664806 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:50.664855 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:50.668948 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.669873 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.669883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.669886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.670089 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.670597 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.670602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.670605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.671098 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:50.671434 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.672033 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.672311 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:05:50.673960 23354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:50.778704 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23231
I20260928 15:05:50.797129 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23365
Sep 28 15:05:50 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:05:50.812666 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.797917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.9.12.194:41555)
0928 15:05:50.798113 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.798118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.798220 (+ 102us) server_negotiation.cc:407] Connection header received
0928 15:05:50.798449 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.798453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.798529 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.798634 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:50.799989 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.800791 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.802073 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.802478 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.802617 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.807007 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:50.807019 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:50.807034 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:50.807071 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:50.809831 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.810676 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.810683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.810685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.810822 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.811216 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.811219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.811220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.811619 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:50.811846 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.812166 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.812405 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:05:50.813539 23488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:50.901211 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23365
I20260928 15:05:50.918467 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23499
Sep 28 15:05:50 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:05:50.933374 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.919272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.9.12.195:42315)
0928 15:05:50.919469 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:50.919475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:50.919494 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:05:50.919680 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:50.919686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:50.919765 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:50.919880 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:50.921243 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.921959 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.923199 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:50.923604 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:50.923734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:50.928065 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:50.928080 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:50.928096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:50.928129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:50.930677 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.931301 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.931314 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.931316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.931446 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:50.931798 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:50.931803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:50.931807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:50.932125 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:50.932683 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:50.932838 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:05:50.933071 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:05:50.934039 23622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:51.018831 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23499
I20260928 15:05:51.034394 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23137
I20260928 15:05:51.035521 23199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:51.147012 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23137
I20260928 15:05:51.168674 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33729
--webserver_interface=127.9.12.254
--webserver_port=43995
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:51.489754 23677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:51.490103 23677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:51.490270 23677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:51.497738 23677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:51.497911 23677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:51.498018 23677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:51.498131 23677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:51.498207 23677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:51.508349 23677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:33729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=43995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:51.509779 23677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:51.511121 23677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:51.534040 23683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:51.534015 23686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:51.534076 23684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:51.534749 23677 server_base.cc:1034] running on GCE node
I20260928 15:05:51.535588 23677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:51.537217 23677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:51.538424 23677 hybrid_clock.cc:648] HybridClock initialized: now 1790607951538386 us; error 31 us; skew 500 ppm
I20260928 15:05:51.538842 23677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:51 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:51.542392 23677 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:51.544090 23677 webserver.cc:460] Webserver started at http://127.9.12.254:43995/ using document root <none> and password file <none>
I20260928 15:05:51.544899 23677 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:51.545059 23677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:51.550911 23677 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:05:51.553761 23693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:51.555078 23677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:05:51.555258 23677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "8f2f821d90a040139832b7d16c06a3a6"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-dmhs"
server_key: "3da9675cc9806d94b1693e88da42fcdb"
server_key_iv: "34a9f116ab0112bc4a2959555be06f08"
server_key_version: "encryptionkey@0"
I20260928 15:05:51.556427 23677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:51.584810 23677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:51.590979 23677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:51.591307 23677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:51.613651 23677 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:33729
I20260928 15:05:51.613672 23744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:33729 every 8 connection(s)
I20260928 15:05:51.615208 23677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:51.621413 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23677
I20260928 15:05:51.622213 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:46463
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:33729
--webserver_port=46275
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:51.623965 23745 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:05:51.628834 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap starting.
I20260928 15:05:51.657915 23745 log.cc:824] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:51.673429 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:51.674068 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap complete.
I20260928 15:05:51.688058 23745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:51.689525 23745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2f821d90a040139832b7d16c06a3a6, State: Initialized, Role: FOLLOWER
I20260928 15:05:51.690353 23745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:51.690697 23745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:51.690846 23745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:51.691078 23745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:05:51.696753 23745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:51.697292 23745 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2f821d90a040139832b7d16c06a3a6; no voters:
I20260928 15:05:51.697952 23745 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:05:51.698086 23749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:05:51.700519 23745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:51.700590 23749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 LEADER]: Becoming Leader. State: Replica: 8f2f821d90a040139832b7d16c06a3a6, State: Running, Role: LEADER
I20260928 15:05:51.701104 23749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:51.703945 23751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2f821d90a040139832b7d16c06a3a6. Latest consensus state: current_term: 2 leader_uuid: "8f2f821d90a040139832b7d16c06a3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } } }
I20260928 15:05:51.704339 23751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:51.704032 23750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f2f821d90a040139832b7d16c06a3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } } }
I20260928 15:05:51.704468 23750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:51.705271 23757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:51.715394 23757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:51.717113 23757 catalog_manager.cc:1259] Loaded cluster ID: f9421391b33e476297b8a7ad83c16b5e
I20260928 15:05:51.717224 23757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:51.721490 23757 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 15:05:51.753031 23757 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 15:05:51.772740 23757 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:51.774566 23757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Loaded TSK: 0
I20260928 15:05:51.776505 23757 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:05:51.994817 23747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:51.995137 23747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:51.995297 23747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:52.002897 23747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:52.003053 23747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:52.003222 23747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:05:52.013195 23747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:46463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=46275
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:52.014545 23747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:52.015831 23747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:52.030875 23772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:52.030895 23773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:52.031791 23775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:52.031976 23747 server_base.cc:1034] running on GCE node
I20260928 15:05:52.032727 23747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:52.033720 23747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:52.034957 23747 hybrid_clock.cc:648] HybridClock initialized: now 1790607952034900 us; error 45 us; skew 500 ppm
I20260928 15:05:52.035284 23747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:52 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:52.038512 23747 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:05:52.040126 23747 webserver.cc:460] Webserver started at http://127.9.12.193:46275/ using document root <none> and password file <none>
I20260928 15:05:52.041016 23747 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:52.041173 23747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:52.046653 23747 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:05:52.049500 23782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.050866 23747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:52.051043 23747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "6c9d5b53831942c0b539f604604fb793"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "b8a51776f49dd29b86400ac81496ed54"
server_key_iv: "1a898eded9e00ee86896956c9f0dc01e"
server_key_version: "encryptionkey@0"
I20260928 15:05:52.051985 23747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:52.067441 23747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:52.071617 23747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:52.071887 23747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:52.072896 23747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:52.074285 23747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:52.074407 23747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.074537 23747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:52.074621 23747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.108457 23747 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:46463
I20260928 15:05:52.108503 23894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:46463 every 8 connection(s)
I20260928 15:05:52.109869 23747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:52.110363 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23747
I20260928 15:05:52.111140 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:41555
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:33729
--webserver_port=32813
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:52 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:52.135701 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.113012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:35921 (local address 127.9.12.254:33729)
0928 15:05:52.113785 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:52.113806 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:52.114576 (+ 770us) server_negotiation.cc:407] Connection header received
0928 15:05:52.116819 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:52.116838 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:52.117287 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:52.117689 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:52.119053 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.120538 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.122952 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.123615 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.123802 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:52.128257 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:52.128288 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:52.128307 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:52.128358 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:52.131333 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:52.132350 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:52.132356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:52.132361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:52.132510 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:52.132990 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:52.132993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:52.132994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:52.133413 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:52.133549 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:52.134475 (+ 926us) server_negotiation.cc:299] Negotiation successful
0928 15:05:52.134780 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":305,"threads_started":1}
I20260928 15:05:52.136368 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.113182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:35921)
0928 15:05:52.114381 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:52.114425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:52.116179 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:52.117785 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:52.117791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:52.118269 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:52.118787 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:52.118803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.120706 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.120714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:52.122809 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:52.122827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.123863 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.123871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:52.124171 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:52.124972 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:52.125004 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:52.127912 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:52.131484 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:52.131495 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:52.131513 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:52.132166 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:52.132621 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:52.132627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:52.132632 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:52.132870 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:52.133559 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:52.133568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:52.133990 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 15:05:52.134310 (+ 320us) client_negotiation.cc:240] Negotiation successful
0928 15:05:52.134934 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":724,"threads_started":1}
I20260928 15:05:52.139297 23895 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33729
I20260928 15:05:52.139609 23895 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:52.140409 23895 heartbeater.cc:511] Master 127.9.12.254:33729 requested a full tablet report, sending...
I20260928 15:05:52.142699 23710 ts_manager.cc:194] Registered new tserver with Master: 6c9d5b53831942c0b539f604604fb793 (127.9.12.193:46463)
I20260928 15:05:52.144831 23710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:35921
W20260928 15:05:52.438119 23899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:52.438369 23899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:52.438479 23899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:52.445653 23899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:52.445811 23899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:52.446053 23899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:05:52.458448 23899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:41555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=32813
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:52.459841 23899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:52.461355 23899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:52.474763 23907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:52.474694 23905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:52.475333 23904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:52.475469 23899 server_base.cc:1034] running on GCE node
I20260928 15:05:52.476155 23899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:52.477109 23899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:52.478344 23899 hybrid_clock.cc:648] HybridClock initialized: now 1790607952478304 us; error 55 us; skew 500 ppm
I20260928 15:05:52.478673 23899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:52 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:52.481745 23899 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:05:52.483213 23899 webserver.cc:460] Webserver started at http://127.9.12.194:32813/ using document root <none> and password file <none>
I20260928 15:05:52.483907 23899 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:52.484062 23899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:52.489283 23899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:05:52.491858 23914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.493216 23899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:05:52.493392 23899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "6e8a3518315c4797907b988921c959de"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "f14655277445d293f91f54a79a1da08c"
server_key_iv: "8ddd12e909d3c3d6de55d20dcf9768b5"
server_key_version: "encryptionkey@0"
I20260928 15:05:52.494360 23899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:52.515270 23899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:52.519395 23899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:52.519691 23899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:52.520757 23899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:52.522163 23899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:52.522289 23899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.522434 23899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:52.522533 23899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.556538 23899 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:41555
I20260928 15:05:52.556674 24026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:41555 every 8 connection(s)
I20260928 15:05:52.557966 23899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:52.567782 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 23899
I20260928 15:05:52.568677 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:42315
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:33729
--webserver_port=36363
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:52 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:52.579648 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.560489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:53117 (local address 127.9.12.254:33729)
0928 15:05:52.560699 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:52.560705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:52.561971 (+ 1266us) server_negotiation.cc:407] Connection header received
0928 15:05:52.563737 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:52.563741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:52.563810 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:52.563898 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:52.565491 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.566192 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.567728 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.568478 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.568608 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:52.572698 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:52.572707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:52.572727 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:52.572775 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:52.576258 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:52.577343 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:52.577347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:52.577349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:52.577481 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:52.577981 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:52.577984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:52.577986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:52.578236 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:52.578345 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:52.579226 (+ 881us) server_negotiation.cc:299] Negotiation successful
0928 15:05:52.579402 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:05:52.581187 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.561089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:53117)
0928 15:05:52.561813 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:52.561857 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:52.563490 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:52.564152 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:52.564174 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:52.564755 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:52.565321 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:52.565339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.566325 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.566329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:52.567580 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:52.567591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:52.568751 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:52.568757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:52.569097 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:52.569827 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:52.569881 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:52.572473 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:52.576472 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:52.576485 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:52.576505 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:52.577215 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:52.577612 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:52.577617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:52.577621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:52.577873 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:52.578357 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:52.578367 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:52.578812 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 15:05:52.579137 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 15:05:52.579711 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":270,"threads_started":1}
I20260928 15:05:52.582639 24027 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33729
I20260928 15:05:52.583017 24027 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:52.583761 24027 heartbeater.cc:511] Master 127.9.12.254:33729 requested a full tablet report, sending...
I20260928 15:05:52.585289 23710 ts_manager.cc:194] Registered new tserver with Master: 6e8a3518315c4797907b988921c959de (127.9.12.194:41555)
I20260928 15:05:52.586172 23710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:53117
W20260928 15:05:52.889816 24030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:52.890080 24030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:52.890296 24030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:52.897043 24030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:52.897141 24030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:52.897325 24030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:05:52.906627 24030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:42315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=36363
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:52.907931 24030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:52.909250 24030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:52.921825 24036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:52.922129 24035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:52.922463 24038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:52.923298 24030 server_base.cc:1034] running on GCE node
I20260928 15:05:52.923933 24030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:52.924861 24030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:52.926079 24030 hybrid_clock.cc:648] HybridClock initialized: now 1790607952926037 us; error 50 us; skew 500 ppm
I20260928 15:05:52.926366 24030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:52 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:52.929483 24030 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:05:52.931035 24030 webserver.cc:460] Webserver started at http://127.9.12.195:36363/ using document root <none> and password file <none>
I20260928 15:05:52.931713 24030 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:52.931862 24030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:52.936990 24030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:05:52.939540 24045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.940869 24030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:05:52.941033 24030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "82b346941c2949d88bbd345c40d5a91f"
format_stamp: "Formatted at 2026-09-28 15:05:50 on dist-test-slave-dmhs"
server_key: "77ba0fe67c5df692cc8f2df151e86f86"
server_key_iv: "c59036e62196523ce8e781590100eafc"
server_key_version: "encryptionkey@0"
I20260928 15:05:52.941962 24030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:52.954648 24030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:52.958988 24030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:52.959278 24030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:52.960327 24030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:52.961661 24030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:52.961768 24030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.961890 24030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:52.961988 24030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:52.991999 24030 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:42315
I20260928 15:05:52.992086 24157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:42315 every 8 connection(s)
I20260928 15:05:52.993501 24030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:52.995467 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24030
Sep 28 15:05:53 dist-test-slave-dmhs krb5kdc[23124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:05:53.011850 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.996430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.9.12.254:33729)
0928 15:05:52.996873 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:52.996880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:52.997281 (+ 401us) server_negotiation.cc:407] Connection header received
0928 15:05:52.997556 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:52.997561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:52.997637 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:52.997805 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:52.999410 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.000420 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.002020 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.002460 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.002548 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.005902 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:53.005910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:53.005915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:53.005962 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:53.009030 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.009743 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.009750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.009754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.009890 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.010290 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.010295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.010298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.010569 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:53.010844 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.011125 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.011414 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":332,"threads_started":1}
I20260928 15:05:53.014849 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.996237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:40899 (local address 127.9.12.254:33729)
0928 15:05:52.996744 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:52.996751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:52.998002 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 15:05:52.999987 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:52.999992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.000061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.000164 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:53.001755 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.002622 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.004164 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.004928 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.005089 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.008977 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:53.008982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:53.008985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:53.009022 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:53.011655 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.012517 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.012522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.012525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.012664 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.013146 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.013151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.013154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.013411 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:53.013533 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.014376 (+ 843us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.014559 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350}
I20260928 15:05:53.015846 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.996965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:40899)
0928 15:05:52.997857 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:52.997899 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:52.999746 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.000479 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.000493 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.000990 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:53.001604 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.001623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.002777 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.002781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.004016 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.004027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.005142 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.005147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.005338 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.005978 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:53.006007 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:53.008766 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:53.011787 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:53.011799 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:53.011818 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:53.012394 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:53.012807 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:53.012811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:53.012814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:53.013019 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:53.013525 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:53.013531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:53.013978 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.014256 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.014777 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":358,"threads_started":1}
I20260928 15:05:53.017132 24158 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33729
I20260928 15:05:53.017421 24158 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:53.018100 24158 heartbeater.cc:511] Master 127.9.12.254:33729 requested a full tablet report, sending...
I20260928 15:05:53.019407 23710 ts_manager.cc:194] Registered new tserver with Master: 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315)
I20260928 15:05:53.020155 23710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:40899
I20260928 15:05:53.026654 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:53.033079 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.027493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.9.12.254:33729)
0928 15:05:53.027644 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.027648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.028112 (+ 464us) server_negotiation.cc:407] Connection header received
0928 15:05:53.028368 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.028371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.028487 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.028622 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:53.029044 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.029741 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.031078 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.031468 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.031590 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.032081 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:53.032537 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:53.032673 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.032792 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.032861 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:05:53.033790 23710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:05:53.040464 23710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:05:53.062880 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.051539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:46463 (local address 127.0.0.1:47944)
0928 15:05:53.053074 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.053107 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.053250 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.054484 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.054491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.054519 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.055001 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.055013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.057217 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.057224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.059890 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.059902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.061767 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.061773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.061967 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.062031 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.062158 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.062361 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":190,"thread_start_us":3025,"threads_started":1}
I20260928 15:05:53.063014 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.051663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.9.12.193:46463)
0928 15:05:53.053952 (+ 2289us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.053963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.053996 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:05:53.054100 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.054106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.054300 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.054475 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.055428 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.057087 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.060041 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.061514 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.061660 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.061893 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.062213 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.062392 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2097,"thread_start_us":1957,"threads_started":1}
I20260928 15:05:53.063910 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.052541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.9.12.195:42315)
0928 15:05:53.054355 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.054365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.054921 (+ 556us) server_negotiation.cc:407] Connection header received
0928 15:05:53.055206 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.055214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.055401 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.055583 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.056785 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.058153 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.060901 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.062007 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.062112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.062341 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.063361 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.063569 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1631,"thread_start_us":1446,"threads_started":1}
I20260928 15:05:53.064041 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.051985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:42315 (local address 127.0.0.1:32938)
0928 15:05:53.054826 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.054842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.055019 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.055991 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.055995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.056018 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.056540 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.056553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.058335 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.058340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.060718 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.060732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.062559 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.062565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.062788 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.062842 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.063545 (+ 703us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.063666 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2716,"mutex_wait_us":4,"thread_start_us":819,"threads_started":1}
I20260928 15:05:53.066030 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.055991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:41555 (local address 127.0.0.1:49094)
0928 15:05:53.056679 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.056695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.056798 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.058153 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.058158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.058185 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.058798 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.058813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.060816 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.060821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.063406 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.063419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.065241 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.065246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.065405 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.065434 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.065534 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.065679 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":383,"threads_started":1}
I20260928 15:05:53.066098 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.055245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.9.12.194:41555)
0928 15:05:53.057630 (+ 2385us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.057638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.057666 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:05:53.057753 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.057760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.057954 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.058136 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.059028 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.060703 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.063553 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.065043 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.065172 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.065385 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.065598 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.065752 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2216,"thread_start_us":2072,"threads_started":1}
I20260928 15:05:53.067301 23830 tablet_service.cc:1467] Processing CreateTablet for tablet 838a434919f54aa391ca3bccd031a1ba (DEFAULT_TABLE table=test-table [id=19a3e9665e7744988cd5c56aefd006dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:53.067871 24093 tablet_service.cc:1467] Processing CreateTablet for tablet 838a434919f54aa391ca3bccd031a1ba (DEFAULT_TABLE table=test-table [id=19a3e9665e7744988cd5c56aefd006dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:53.068846 23830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838a434919f54aa391ca3bccd031a1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:53.069243 24093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838a434919f54aa391ca3bccd031a1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:53.070024 23962 tablet_service.cc:1467] Processing CreateTablet for tablet 838a434919f54aa391ca3bccd031a1ba (DEFAULT_TABLE table=test-table [id=19a3e9665e7744988cd5c56aefd006dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:53.071369 23962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838a434919f54aa391ca3bccd031a1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:53.081121 24174 tablet_bootstrap.cc:492] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Bootstrap starting.
I20260928 15:05:53.084692 24174 tablet_bootstrap.cc:654] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:53.085542 24175 tablet_bootstrap.cc:492] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Bootstrap starting.
I20260928 15:05:53.086012 24174 log.cc:824] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:53.087404 24176 tablet_bootstrap.cc:492] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Bootstrap starting.
I20260928 15:05:53.089286 24174 tablet_bootstrap.cc:492] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: No bootstrap required, opened a new log
I20260928 15:05:53.089345 24175 tablet_bootstrap.cc:654] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:53.089570 24174 ts_tablet_manager.cc:1397] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:05:53.090557 24175 log.cc:824] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:53.090945 24176 tablet_bootstrap.cc:654] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:53.092041 24176 log.cc:824] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:53.094475 24176 tablet_bootstrap.cc:492] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: No bootstrap required, opened a new log
I20260928 15:05:53.094734 24176 ts_tablet_manager.cc:1397] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 15:05:53.098940 24175 tablet_bootstrap.cc:492] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: No bootstrap required, opened a new log
I20260928 15:05:53.099203 24175 ts_tablet_manager.cc:1397] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 15:05:53.101266 24174 raft_consensus.cc:348] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.101724 24174 raft_consensus.cc:374] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:53.101814 24174 raft_consensus.cc:729] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b346941c2949d88bbd345c40d5a91f, State: Initialized, Role: FOLLOWER
I20260928 15:05:53.102425 24174 consensus_queue.cc:260] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.103029 24176 raft_consensus.cc:348] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.103482 24176 raft_consensus.cc:374] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:53.103595 24176 raft_consensus.cc:729] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e8a3518315c4797907b988921c959de, State: Initialized, Role: FOLLOWER
I20260928 15:05:53.103699 24158 heartbeater.cc:503] Master 127.9.12.254:33729 was elected leader, sending a full tablet report...
I20260928 15:05:53.104206 24176 consensus_queue.cc:260] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.105391 24174 ts_tablet_manager.cc:1428] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 15:05:53.108762 24176 ts_tablet_manager.cc:1428] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260928 15:05:53.108938 24027 heartbeater.cc:503] Master 127.9.12.254:33729 was elected leader, sending a full tablet report...
I20260928 15:05:53.112221 24175 raft_consensus.cc:348] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.112716 24175 raft_consensus.cc:374] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:53.112836 24175 raft_consensus.cc:729] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9d5b53831942c0b539f604604fb793, State: Initialized, Role: FOLLOWER
I20260928 15:05:53.113502 24175 consensus_queue.cc:260] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.114549 23895 heartbeater.cc:503] Master 127.9.12.254:33729 was elected leader, sending a full tablet report...
I20260928 15:05:53.115182 24175 ts_tablet_manager.cc:1428] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260928 15:05:53.245265 24159 tablet.cc:2369] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:53.297348 24181 raft_consensus.cc:482] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:53.297674 24181 raft_consensus.cc:504] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.299504 24181 leader_election.cc:288] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9d5b53831942c0b539f604604fb793 (127.9.12.193:46463), 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315)
I20260928 15:05:53.300940 24182 raft_consensus.cc:482] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:53.301256 24182 raft_consensus.cc:504] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
W20260928 15:05:53.309932 24028 tablet.cc:2369] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:53.310003 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.299933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:48319 (local address 127.9.12.193:46463)
0928 15:05:53.300209 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.300218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.301049 (+ 831us) server_negotiation.cc:407] Connection header received
0928 15:05:53.301355 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.301359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.301464 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.301576 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.302366 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.303180 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.306514 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.308396 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.308557 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.308671 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.309429 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.309619 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 15:05:53.310545 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.300018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:51075 (local address 127.9.12.195:42315)
0928 15:05:53.300279 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.300289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.301194 (+ 905us) server_negotiation.cc:407] Connection header received
0928 15:05:53.301462 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.301468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.301546 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.301626 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.302540 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.303591 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.307436 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.309250 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.309415 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.309520 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.310161 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.310318 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:05:53.310964 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.299698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:46463 (local address 127.9.12.194:48319)
0928 15:05:53.300922 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.300939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.301191 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.301610 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.301618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.301645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.302197 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.302210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.303363 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.303372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.306280 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.306301 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.308916 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.308926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.309192 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.309246 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.310249 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.310454 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":5,"thread_start_us":1026,"threads_started":1}
I20260928 15:05:53.311618 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.299983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:42315 (local address 127.9.12.194:51075)
0928 15:05:53.301089 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.301107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.301306 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.301869 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.301874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.301907 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.302362 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.302373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.303766 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.303771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.307218 (+ 3447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.307236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.309739 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.309745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.309961 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.309997 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.310890 (+ 893us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.311103 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":691,"thread_start_us":496,"threads_started":1}
I20260928 15:05:53.311993 24182 leader_election.cc:288] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e8a3518315c4797907b988921c959de (127.9.12.194:41555), 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315)
I20260928 15:05:53.312197 23850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838a434919f54aa391ca3bccd031a1ba" candidate_uuid: "6e8a3518315c4797907b988921c959de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9d5b53831942c0b539f604604fb793" is_pre_election: true
I20260928 15:05:53.312798 23850 raft_consensus.cc:2454] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e8a3518315c4797907b988921c959de in term 0.
I20260928 15:05:53.313601 23918 leader_election.cc:302] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9d5b53831942c0b539f604604fb793, 6e8a3518315c4797907b988921c959de; no voters:
I20260928 15:05:53.314090 24181 raft_consensus.cc:2781] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:53.313967 24113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838a434919f54aa391ca3bccd031a1ba" candidate_uuid: "6e8a3518315c4797907b988921c959de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b346941c2949d88bbd345c40d5a91f" is_pre_election: true
I20260928 15:05:53.314241 24181 raft_consensus.cc:482] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:53.314339 24181 raft_consensus.cc:3037] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:53.314608 24113 raft_consensus.cc:2454] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e8a3518315c4797907b988921c959de in term 0.
I20260928 15:05:53.317149 24181 raft_consensus.cc:504] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.318324 23850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838a434919f54aa391ca3bccd031a1ba" candidate_uuid: "6e8a3518315c4797907b988921c959de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9d5b53831942c0b539f604604fb793"
I20260928 15:05:53.318331 24113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838a434919f54aa391ca3bccd031a1ba" candidate_uuid: "6e8a3518315c4797907b988921c959de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b346941c2949d88bbd345c40d5a91f"
I20260928 15:05:53.318538 23850 raft_consensus.cc:3037] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:53.318560 24113 raft_consensus.cc:3037] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:53.319561 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.303887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:44311 (local address 127.9.12.194:41555)
0928 15:05:53.304106 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.304112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.306214 (+ 2102us) server_negotiation.cc:407] Connection header received
0928 15:05:53.312360 (+ 6146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.312385 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.312457 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.312527 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.313423 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.314506 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.317124 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.318359 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.318522 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.318602 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.319198 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.319322 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:05:53.320299 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.303532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:41555 (local address 127.9.12.193:44311)
0928 15:05:53.306097 (+ 2565us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.306121 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.312142 (+ 6021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.312756 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.312764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.312809 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.313269 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.313284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.314687 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.314692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.316944 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.316958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.318761 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.318767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.318975 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.319024 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.319705 (+ 681us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.319857 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"mutex_wait_us":3,"spinlock_wait_cycles":5376,"thread_start_us":8780,"threads_started":1}
I20260928 15:05:53.320549 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.304270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:54869 (local address 127.9.12.195:42315)
0928 15:05:53.310773 (+ 6503us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.310779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.312776 (+ 1997us) server_negotiation.cc:407] Connection header received
0928 15:05:53.312986 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.312990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.313065 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.313159 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:53.314022 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.315165 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.317597 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.318978 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.319158 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.319250 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.319932 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.320077 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6387,"thread_start_us":7528,"threads_started":1}
I20260928 15:05:53.320567 24181 leader_election.cc:288] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [CANDIDATE]: Term 1 election: Requested vote from peers 6c9d5b53831942c0b539f604604fb793 (127.9.12.193:46463), 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315)
I20260928 15:05:53.321213 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.303731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:42315 (local address 127.9.12.193:54869)
0928 15:05:53.312669 (+ 8938us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.312687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:53.312832 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:53.313376 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:53.313382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:53.313410 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:53.313751 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.313762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.315454 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.315459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.317443 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:53.317459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.319407 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.319416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:53.319592 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.319775 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 15:05:53.320715 (+ 940us) client_negotiation.cc:240] Negotiation successful
0928 15:05:53.320851 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8814,"spinlock_wait_cycles":1408,"thread_start_us":1960,"threads_started":1}
I20260928 15:05:53.321843 23850 raft_consensus.cc:2454] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e8a3518315c4797907b988921c959de in term 1.
I20260928 15:05:53.321720 23982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838a434919f54aa391ca3bccd031a1ba" candidate_uuid: "6c9d5b53831942c0b539f604604fb793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e8a3518315c4797907b988921c959de" is_pre_election: true
I20260928 15:05:53.321928 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838a434919f54aa391ca3bccd031a1ba" candidate_uuid: "6c9d5b53831942c0b539f604604fb793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b346941c2949d88bbd345c40d5a91f" is_pre_election: true
I20260928 15:05:53.322263 23982 raft_consensus.cc:2379] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c9d5b53831942c0b539f604604fb793 in current term 1: Already voted for candidate 6e8a3518315c4797907b988921c959de in this term.
I20260928 15:05:53.322535 23918 leader_election.cc:302] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9d5b53831942c0b539f604604fb793, 6e8a3518315c4797907b988921c959de; no voters:
I20260928 15:05:53.322666 24113 raft_consensus.cc:2454] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e8a3518315c4797907b988921c959de in term 1.
I20260928 15:05:53.322906 24181 raft_consensus.cc:2781] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:53.323206 23784 leader_election.cc:302] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c9d5b53831942c0b539f604604fb793; no voters: 6e8a3518315c4797907b988921c959de, 82b346941c2949d88bbd345c40d5a91f
I20260928 15:05:53.323498 24181 raft_consensus.cc:686] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 1 LEADER]: Becoming Leader. State: Replica: 6e8a3518315c4797907b988921c959de, State: Running, Role: LEADER
I20260928 15:05:53.323588 24182 raft_consensus.cc:2726] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:05:53.324461 24181 consensus_queue.cc:237] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } }
I20260928 15:05:53.329264 23709 catalog_manager.cc:5495] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e8a3518315c4797907b988921c959de (127.9.12.194). New cstate: current_term: 1 leader_uuid: "6e8a3518315c4797907b988921c959de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e8a3518315c4797907b988921c959de" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 41555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 } health_report { overall_health: UNKNOWN } } }
I20260928 15:05:53.357692 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.352471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.9.12.194:41555)
0928 15:05:53.352641 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.352646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.352663 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:05:53.352828 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.352832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.352923 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.353039 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:53.353344 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.354056 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.355459 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.355972 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.356082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.356624 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:53.357134 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:53.357238 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.357312 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.357373 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 15:05:53.362316 23896 tablet.cc:2369] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:53.365200 24113 raft_consensus.cc:1261] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Refusing update from remote peer 6e8a3518315c4797907b988921c959de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:53.365340 23850 raft_consensus.cc:1261] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Refusing update from remote peer 6e8a3518315c4797907b988921c959de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:53.366080 24188 consensus_queue.cc:1035] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b346941c2949d88bbd345c40d5a91f" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:53.366436 24181 consensus_queue.cc:1035] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9d5b53831942c0b539f604604fb793" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:53.378365 24191 mvcc.cc:205] Tried to move back new op lower bound from 7334330176975568896 to 7334330176821145600. Current Snapshot: MvccSnapshot[applied={T|T < 7334330176975568896}]
I20260928 15:05:53.380600 24192 mvcc.cc:205] Tried to move back new op lower bound from 7334330176975568896 to 7334330176821145600. Current Snapshot: MvccSnapshot[applied={T|T < 7334330176975568896 or (T in {7334330176975568896})}]
I20260928 15:05:53.390703 23710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:53.390935 23710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:53.394121 23710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Sending DeleteTablet for 3 replicas of tablet 838a434919f54aa391ca3bccd031a1ba
I20260928 15:05:53.395321 24093 tablet_service.cc:1514] Processing DeleteTablet for tablet 838a434919f54aa391ca3bccd031a1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:53 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:32938
I20260928 15:05:53.395591 23962 tablet_service.cc:1514] Processing DeleteTablet for tablet 838a434919f54aa391ca3bccd031a1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:53 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:49094
I20260928 15:05:53.395748 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23747
I20260928 15:05:53.396102 23830 tablet_service.cc:1514] Processing DeleteTablet for tablet 838a434919f54aa391ca3bccd031a1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:53 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:47944
I20260928 15:05:53.397783 24201 tablet_replica.cc:330] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: stopping tablet replica
I20260928 15:05:53.398237 24201 raft_consensus.cc:2229] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:53.398918 24202 tablet_replica.cc:330] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: stopping tablet replica
I20260928 15:05:53.399379 24202 raft_consensus.cc:2229] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:05:53.400279 24202 raft_consensus.cc:2258] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:53.400897 24201 raft_consensus.cc:2258] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:53.403364 24203 tablet_replica.cc:330] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: stopping tablet replica
I20260928 15:05:53.404430 24203 raft_consensus.cc:2229] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:53.404515 24201 ts_tablet_manager.cc:1905] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:53.404603 24202 ts_tablet_manager.cc:1905] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:53.404870 24203 raft_consensus.cc:2258] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:53.406677 24203 ts_tablet_manager.cc:1905] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:53.411283 24201 ts_tablet_manager.cc:1918] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:53.411425 24201 log.cc:1196] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/838a434919f54aa391ca3bccd031a1ba
I20260928 15:05:53.411757 24202 ts_tablet_manager.cc:1918] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:53.411898 24202 log.cc:1196] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/838a434919f54aa391ca3bccd031a1ba
I20260928 15:05:53.411927 24201 ts_tablet_manager.cc:1939] T 838a434919f54aa391ca3bccd031a1ba P 82b346941c2949d88bbd345c40d5a91f: Deleting consensus metadata
I20260928 15:05:53.412472 24202 ts_tablet_manager.cc:1939] T 838a434919f54aa391ca3bccd031a1ba P 6e8a3518315c4797907b988921c959de: Deleting consensus metadata
I20260928 15:05:53.413731 24203 ts_tablet_manager.cc:1918] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:53.413750 23695 catalog_manager.cc:4841] TS 82b346941c2949d88bbd345c40d5a91f (127.9.12.195:42315): tablet 838a434919f54aa391ca3bccd031a1ba (table test-table [id=19a3e9665e7744988cd5c56aefd006dc]) successfully deleted
I20260928 15:05:53.413905 24203 log.cc:1196] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/838a434919f54aa391ca3bccd031a1ba
I20260928 15:05:53.414321 23697 catalog_manager.cc:4841] TS 6e8a3518315c4797907b988921c959de (127.9.12.194:41555): tablet 838a434919f54aa391ca3bccd031a1ba (table test-table [id=19a3e9665e7744988cd5c56aefd006dc]) successfully deleted
I20260928 15:05:53.414453 24203 ts_tablet_manager.cc:1939] T 838a434919f54aa391ca3bccd031a1ba P 6c9d5b53831942c0b539f604604fb793: Deleting consensus metadata
I20260928 15:05:53.415617 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.396599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.9.12.193:46463)
0928 15:05:53.396762 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.396768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.396845 (+ 77us) server_negotiation.cc:407] Connection header received
0928 15:05:53.397052 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.397057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.397181 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.397281 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:53.399171 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.400505 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.402665 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.403371 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.403502 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.407171 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:53.407189 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:53.407208 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:53.407256 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:53.411168 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.412330 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.412342 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.412348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.412520 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.413112 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.413117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.413120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.413619 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:53.414825 (+ 1206us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.415010 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.415245 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:05:53.416160 23697 catalog_manager.cc:4841] TS 6c9d5b53831942c0b539f604604fb793 (127.9.12.193:46463): tablet 838a434919f54aa391ca3bccd031a1ba (table test-table [id=19a3e9665e7744988cd5c56aefd006dc]) successfully deleted
I20260928 15:05:53.416226 23890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:53.507799 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23747
I20260928 15:05:53.526681 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23899
I20260928 15:05:53.544986 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.527550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.9.12.194:41555)
0928 15:05:53.527754 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.527761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.527783 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:05:53.528026 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.528032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.528138 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.528247 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:53.529640 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.530429 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.531837 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.532378 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.532519 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.539062 (+ 6543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:53.539104 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:53.539119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:53.539165 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:53.542204 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.542878 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.542887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.542891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.543024 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.543415 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.543420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.543424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.543812 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:53.544162 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.544431 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.544675 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:05:53.545775 24022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:53.658569 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23899
I20260928 15:05:53.677524 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24030
I20260928 15:05:53.696237 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.678336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.9.12.195:42315)
0928 15:05:53.678536 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:53.678543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:53.678579 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:05:53.678805 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:53.678809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:53.678883 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:53.678985 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:53.680441 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.681108 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.682402 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:53.682797 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:53.682928 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:53.690874 (+ 7946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:53.690892 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:53.690908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:53.690942 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:53.693525 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.694207 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.694214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.694217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.694335 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:53.694710 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:53.694717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:53.694720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:53.695090 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:53.695388 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:53.695696 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 15:05:53.695916 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:05:53.696961 24153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:53.788594 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24030
I20260928 15:05:53.804711 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 23677
I20260928 15:05:53.805859 23740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:05:53.915138 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 23677
I20260928 15:05:53.937458 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33729
--webserver_interface=127.9.12.254
--webserver_port=43995
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:54.252861 24208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:54.253113 24208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:54.253288 24208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:54.260517 24208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:54.260638 24208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:54.260725 24208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:54.260813 24208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:54.260885 24208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:54.270329 24208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:33729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=43995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:54.271675 24208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:54.273113 24208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:54.284593 24213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:54.284651 24214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:54.284502 24216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:54.285689 24208 server_base.cc:1034] running on GCE node
I20260928 15:05:54.286371 24208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:54.287654 24208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:54.288842 24208 hybrid_clock.cc:648] HybridClock initialized: now 1790607954288793 us; error 35 us; skew 500 ppm
I20260928 15:05:54.289117 24208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:54 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607954, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:54.292001 24208 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:05:54.293503 24208 webserver.cc:460] Webserver started at http://127.9.12.254:43995/ using document root <none> and password file <none>
I20260928 15:05:54.294144 24208 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:54.294278 24208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:54.299901 24208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:05:54.302685 24223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:54.303902 24208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:05:54.304075 24208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "8f2f821d90a040139832b7d16c06a3a6"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-dmhs"
server_key: "3da9675cc9806d94b1693e88da42fcdb"
server_key_iv: "34a9f116ab0112bc4a2959555be06f08"
server_key_version: "encryptionkey@0"
I20260928 15:05:54.305119 24208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:54.321393 24208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:54.325054 24208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:54.325313 24208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:54.343962 24208 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:33729
I20260928 15:05:54.343973 24274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:33729 every 8 connection(s)
I20260928 15:05:54.345553 24208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:54.349527 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24208
I20260928 15:05:54.350559 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:46463
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:33729
--webserver_port=46275
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:54.353528 24275 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:05:54.358309 24275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap starting.
I20260928 15:05:54.378383 24275 log.cc:824] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:54.393642 24275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:54.394138 24275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: Bootstrap complete.
I20260928 15:05:54.403122 24275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:54.404007 24275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2f821d90a040139832b7d16c06a3a6, State: Initialized, Role: FOLLOWER
I20260928 15:05:54.404718 24275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:54.404999 24275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:54.405153 24275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:54.405324 24275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:05:54.409327 24275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:54.409811 24275 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2f821d90a040139832b7d16c06a3a6; no voters:
I20260928 15:05:54.410523 24275 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:05:54.410665 24279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:05:54.412798 24275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:54.417683 24279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [term 3 LEADER]: Becoming Leader. State: Replica: 8f2f821d90a040139832b7d16c06a3a6, State: Running, Role: LEADER
I20260928 15:05:54.418376 24279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } }
I20260928 15:05:54.422288 24280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8f2f821d90a040139832b7d16c06a3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } } }
I20260928 15:05:54.422614 24280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:54.432857 24281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2f821d90a040139832b7d16c06a3a6. Latest consensus state: current_term: 3 leader_uuid: "8f2f821d90a040139832b7d16c06a3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2f821d90a040139832b7d16c06a3a6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33729 } } }
I20260928 15:05:54.433137 24281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:54.437793 24293 catalog_manager.cc:1259] Loaded cluster ID: f9421391b33e476297b8a7ad83c16b5e
I20260928 15:05:54.437892 24293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: loading cluster ID for follower catalog manager: success
W20260928 15:05:54.439813 24293 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8f2f821d90a040139832b7d16c06a3a6: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260928 15:05:54.439949 24293 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260928 15:05:54.440076 24294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:54.445088 24294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:54.446231 24294 catalog_manager.cc:1259] Loaded cluster ID: f9421391b33e476297b8a7ad83c16b5e
I20260928 15:05:54.446314 24294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 15:05:54.447916 24294 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4ec15239ad google::LogMessage::Fail() at ??:0
@ 0x7f4ec152688a google::LogMessage::SendToLog() at ??:0
@ 0x7f4ec15234de google::LogMessage::Flush() at ??:0
@ 0x7f4ec1524a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4ed6f56512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4ed6f53976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4ed6fdb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4ed6fdb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4ec416f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4ec417291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4ec4172799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4ec414deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4ec74276db start_thread at ??:0
@ 0x7f4ec03bd61f clone at ??:0
W20260928 15:05:54.707612 24277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:54.707932 24277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:54.708106 24277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:54.715283 24277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:54.715466 24277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:54.715660 24277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:05:54.725634 24277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:46463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=46275
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:54.727046 24277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:54.728577 24277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:54.742600 24316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:54.743244 24317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:54.744653 24319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:54.746133 24277 server_base.cc:1034] running on GCE node
I20260928 15:05:54.746685 24277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:54.747641 24277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:54.748893 24277 hybrid_clock.cc:648] HybridClock initialized: now 1790607954748833 us; error 52 us; skew 500 ppm
I20260928 15:05:54.749243 24277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:54 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607954, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:54.752460 24277 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:05:54.753964 24277 webserver.cc:460] Webserver started at http://127.9.12.193:46275/ using document root <none> and password file <none>
I20260928 15:05:54.754688 24277 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:54.754813 24277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:54.760466 24277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:05:54.763079 24326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:54.764454 24277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:05:54.764648 24277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "6c9d5b53831942c0b539f604604fb793"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "b8a51776f49dd29b86400ac81496ed54"
server_key_iv: "1a898eded9e00ee86896956c9f0dc01e"
server_key_version: "encryptionkey@0"
I20260928 15:05:54.765643 24277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:54.805377 24277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:54.809618 24277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:54.809918 24277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:54.810927 24277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:54.812256 24277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:54.812386 24277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:54.812536 24277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:54.812642 24277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:54.845207 24277 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:46463
I20260928 15:05:54.845292 24438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:46463 every 8 connection(s)
I20260928 15:05:54.846623 24277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:54.847195 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24277
I20260928 15:05:54.848001 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:41555
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:33729
--webserver_port=32813
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:54.852068 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:54.849729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:46155)
0928 15:05:54.850631 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:54.850881 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
W20260928 15:05:54.852793 24439 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:33729 (0 consecutive failures): Network error: Failed to ping master at 127.9.12.254:33729: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
I20260928 15:05:54.853664 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:54.853364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:44709)
0928 15:05:54.853501 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:54.853565 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:05:54.854519 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:54.854229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:51165)
0928 15:05:54.854350 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:54.854417 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:05:54.854775 24439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:05:55.173463 24442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:55.173748 24442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:55.173933 24442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:55.181306 24442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:55.181432 24442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:55.181605 24442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:05:55.191124 24442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:41555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=32813
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:55.192378 24442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:55.193650 24442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:55.207140 24447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:55.207306 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:55.207109 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:55.207763 24442 server_base.cc:1034] running on GCE node
I20260928 15:05:55.208633 24442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:55.209590 24442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:55.210788 24442 hybrid_clock.cc:648] HybridClock initialized: now 1790607955210738 us; error 42 us; skew 500 ppm
I20260928 15:05:55.211110 24442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:55 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607955, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:55.214222 24442 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:05:55.215706 24442 webserver.cc:460] Webserver started at http://127.9.12.194:32813/ using document root <none> and password file <none>
I20260928 15:05:55.216459 24442 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:55.216566 24442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:55.221613 24442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:05:55.226023 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:55.227347 24442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:05:55.227528 24442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "6e8a3518315c4797907b988921c959de"
format_stamp: "Formatted at 2026-09-28 15:05:49 on dist-test-slave-dmhs"
server_key: "f14655277445d293f91f54a79a1da08c"
server_key_iv: "8ddd12e909d3c3d6de55d20dcf9768b5"
server_key_version: "encryptionkey@0"
I20260928 15:05:55.228564 24442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:55.252727 24442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:55.256786 24442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:55.257084 24442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:55.258124 24442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:55.259451 24442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:55.259583 24442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:55.259720 24442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:55.259814 24442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:55.293344 24442 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:41555
I20260928 15:05:55.293431 24569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:41555 every 8 connection(s)
I20260928 15:05:55.294863 24442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:55.299662 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.297867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:58887)
0928 15:05:55.298576 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.298802 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":274,"threads_started":1}
W20260928 15:05:55.300356 24570 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:33729 (0 consecutive failures): Network error: Failed to ping master at 127.9.12.254:33729: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
I20260928 15:05:55.301122 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.300821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:46833)
0928 15:05:55.300948 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.301020 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 15:05:55.301894 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.301624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:39665)
0928 15:05:55.301732 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.301799 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:05:55.302165 24570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:05:55.302909 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24442
I20260928 15:05:55.303539 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:42315
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:33729
--webserver_port=36363
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:55.615375 24573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:55.615628 24573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:55.615717 24573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:55.622532 24573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:55.622623 24573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:55.622725 24573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:05:55.632254 24573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:42315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=36363
--tserver_master_addrs=127.9.12.254:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:05:55.633572 24573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:55.635233 24573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:55.648176 24579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:55.648152 24578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:55.648214 24581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:55.649022 24573 server_base.cc:1034] running on GCE node
I20260928 15:05:55.649655 24573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:55.650612 24573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:55.651836 24573 hybrid_clock.cc:648] HybridClock initialized: now 1790607955651797 us; error 50 us; skew 500 ppm
I20260928 15:05:55.652149 24573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:05:55 dist-test-slave-dmhs krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607955, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:55.655241 24573 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:05:55.656760 24573 webserver.cc:460] Webserver started at http://127.9.12.195:36363/ using document root <none> and password file <none>
I20260928 15:05:55.657403 24573 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:55.657550 24573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:55.662482 24573 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 15:05:55.665040 24588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:55.666229 24573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:05:55.666374 24573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "82b346941c2949d88bbd345c40d5a91f"
format_stamp: "Formatted at 2026-09-28 15:05:50 on dist-test-slave-dmhs"
server_key: "77ba0fe67c5df692cc8f2df151e86f86"
server_key_iv: "c59036e62196523ce8e781590100eafc"
server_key_version: "encryptionkey@0"
I20260928 15:05:55.667297 24573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:55.685765 24573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:55.690109 24573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:55.690395 24573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:55.691365 24573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:55.692700 24573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:55.692823 24573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:55.692955 24573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:55.693040 24573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:55.724136 24573 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:42315
I20260928 15:05:55.724221 24700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:42315 every 8 connection(s)
I20260928 15:05:55.725739 24573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:55.730381 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.728677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:42085)
0928 15:05:55.729387 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.729587 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":268,"threads_started":1}
W20260928 15:05:55.731068 24701 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:33729 (0 consecutive failures): Network error: Failed to ping master at 127.9.12.254:33729: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
I20260928 15:05:55.731746 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.731486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:40457)
0928 15:05:55.731601 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.731658 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 15:05:55.732602 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.732309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:48947)
0928 15:05:55.732411 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.732480 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 15:05:55.732906 24701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:05:55.735015 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24573
I20260928 15:05:55.856405 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.855546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:47781)
0928 15:05:55.856007 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.856121 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260928 15:05:56.303409 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:56.302644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:42413)
0928 15:05:56.303098 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:56.303180 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260928 15:05:56.734241 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:56.733453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:38261)
0928 15:05:56.733898 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:56.733999 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":217,"threads_started":1}
I20260928 15:05:56.858089 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:56.857346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:54073)
0928 15:05:56.857819 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:56.857914 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":226,"threads_started":1}
I20260928 15:05:57.305235 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:57.304549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:57739)
0928 15:05:57.304985 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:57.305076 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":210,"threads_started":1}
I20260928 15:05:57.735769 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:57.735098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:42993)
0928 15:05:57.735524 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:57.735606 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260928 15:05:57.859712 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:57.859009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:43909)
0928 15:05:57.859418 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:57.859541 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":213,"threads_started":1}
I20260928 15:05:58.306980 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:58.306192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:33059)
0928 15:05:58.306709 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:58.306787 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":320,"threads_started":1}
I20260928 15:05:58.737447 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:58.736748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:48529)
0928 15:05:58.737192 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:58.737265 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":250,"threads_started":1}
I20260928 15:05:58.861501 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:58.860681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:53425)
0928 15:05:58.861245 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:58.861321 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":312,"threads_started":1}
I20260928 15:05:59.308808 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:59.307904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:48539)
0928 15:05:59.308459 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:59.308588 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":230,"threads_started":1}
I20260928 15:05:59.739107 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:59.738352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:34023)
0928 15:05:59.738821 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:59.738905 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260928 15:05:59.863224 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:59.862433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:35959)
0928 15:05:59.862967 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:59.863052 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":282,"threads_started":1}
I20260928 15:06:00.310734 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:00.309858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:59195)
0928 15:06:00.310479 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:00.310561 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":314,"threads_started":1}
I20260928 15:06:00.741204 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:00.740172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:51269)
0928 15:06:00.740925 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:00.741035 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":465,"threads_started":1}
I20260928 15:06:00.864926 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:00.864134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:58253)
0928 15:06:00.864619 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:00.864708 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":275,"threads_started":1}
I20260928 15:06:01.312486 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:01.311689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:44503)
0928 15:06:01.312154 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:01.312250 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
I20260928 15:06:01.742848 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:01.742146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:53033)
0928 15:06:01.742602 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:01.742693 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260928 15:06:01.866605 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:01.865761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:38657)
0928 15:06:01.866263 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:01.866343 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":227,"threads_started":1}
I20260928 15:06:02.314342 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:02.313537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:50607)
0928 15:06:02.314057 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:02.314143 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":203,"threads_started":1}
I20260928 15:06:02.744824 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:02.743982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:39985)
0928 15:06:02.744526 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:02.744612 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260928 15:06:02.868352 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:02.867608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:56897)
0928 15:06:02.868056 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:02.868139 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260928 15:06:03.316047 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:03.315362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:41235)
0928 15:06:03.315802 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:03.315883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260928 15:06:03.746639 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:03.745904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:55659)
0928 15:06:03.746375 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:03.746462 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
I20260928 15:06:03.870398 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:03.869408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:43273)
0928 15:06:03.869992 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:03.870093 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":326,"threads_started":1}
I20260928 15:06:04.317996 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:04.317177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:38295)
0928 15:06:04.317703 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:04.317792 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":262,"threads_started":1}
I20260928 15:06:04.748312 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:04.747556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:44367)
0928 15:06:04.748031 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:04.748108 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260928 15:06:04.871914 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:04.871247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:55443)
0928 15:06:04.871669 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:04.871743 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":241,"threads_started":1}
I20260928 15:06:05.319887 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:05.319099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:40119)
0928 15:06:05.319616 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:05.319724 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":303,"threads_started":1}
I20260928 15:06:05.749989 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:05.749202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:53767)
0928 15:06:05.749684 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:05.749768 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":265,"threads_started":1}
I20260928 15:06:05.873574 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:05.872854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:52503)
0928 15:06:05.873289 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:05.873386 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260928 15:06:06.321523 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.320813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:40619)
0928 15:06:06.321239 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.321334 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260928 15:06:06.751736 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.750971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:53757)
0928 15:06:06.751467 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.751558 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
I20260928 15:06:06.875185 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.874495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:50043)
0928 15:06:06.874948 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.875029 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260928 15:06:07.323184 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.322496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:43113)
0928 15:06:07.322934 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:07.323026 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":214,"threads_started":1}
I20260928 15:06:07.753428 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.752760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:48165)
0928 15:06:07.753196 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:07.753277 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":208,"threads_started":1}
I20260928 15:06:07.877007 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.876094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:34177)
0928 15:06:07.876690 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:07.876803 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":320,"threads_started":1}
I20260928 15:06:08.324822 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:08.324036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:46483)
0928 15:06:08.324550 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:08.324641 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":281,"threads_started":1}
I20260928 15:06:08.755198 24743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:08.754358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:57261)
0928 15:06:08.754827 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:08.754925 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":224,"threads_started":1}
I20260928 15:06:08.878643 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:08.877954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:41247)
0928 15:06:08.878385 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:08.878466 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260928 15:06:09.326382 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:09.325651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:38657)
0928 15:06:09.326115 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:09.326198 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260928 15:06:09.757030 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:09.756191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:47423)
0928 15:06:09.756758 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:09.756840 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":353,"threads_started":1}
I20260928 15:06:09.880769 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:09.879918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.193:47007)
0928 15:06:09.880473 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:09.880568 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":313,"threads_started":1}
I20260928 15:06:10.328234 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:10.327478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.194:46795)
0928 15:06:10.327922 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:10.328037 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":220,"threads_started":1}
I20260928 15:06:10.739846 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24277
I20260928 15:06:10.758811 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:10.758075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33729 (local address 127.9.12.195:53525)
0928 15:06:10.758540 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:10.758636 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260928 15:06:10.764369 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:10.740995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.9.12.193:46463)
0928 15:06:10.741527 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:10.741544 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:10.741588 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:06:10.744001 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:10.744017 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:10.744509 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:10.744799 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:10.746203 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:10.747417 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:10.748791 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:10.749444 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:10.749575 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:10.758157 (+ 8582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:10.758171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:10.758194 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:10.758271 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:10.761368 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:10.762036 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:10.762043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:10.762050 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:10.762205 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:10.762528 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:10.762531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:10.762533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:10.763029 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:10.763258 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:10.763594 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:06:10.764037 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":242,"threads_started":1}
I20260928 15:06:10.765770 24434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:10.848649 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24277
I20260928 15:06:10.866268 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24442
I20260928 15:06:10.884197 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:10.867288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.9.12.194:41555)
0928 15:06:10.867837 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:10.867850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:10.867888 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:06:10.869493 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:10.869503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:10.869919 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:10.870299 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:10.872116 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:10.873444 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:10.874615 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:10.875302 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:10.875503 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:10.878124 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:10.878131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:10.878147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:10.878182 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:10.881232 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:10.881965 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:10.881971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:10.881977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:10.882134 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:10.882501 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:10.882503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:10.882505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:10.882962 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:10.883172 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:10.883530 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 15:06:10.883929 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260928 15:06:10.885529 24565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:10.970566 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24442
I20260928 15:06:10.989548 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24573
I20260928 15:06:11.007639 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:10.990643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.9.12.195:42315)
0928 15:06:10.991244 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:10.991258 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:10.991298 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:06:10.992990 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:10.992998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:10.993425 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:10.993780 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:10.995237 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:10.996575 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:10.997863 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:10.998526 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:10.998745 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:11.001413 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:11.001420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:11.001435 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:11.001473 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:11.004212 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:11.004972 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:11.004985 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:11.004992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:11.005167 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:11.005580 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:11.005587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:11.005590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:11.006109 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:11.006438 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:11.006893 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 15:06:11.007338 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
I20260928 15:06:11.009084 24696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:11.092190 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24573
2026-09-28T15:06:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24166 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:06:11 dist-test-slave-dmhs krb5kdc[24758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:06:11 dist-test-slave-dmhs krb5kdc[24758](info): set up 2 sockets
Sep 28 15:06:11 dist-test-slave-dmhs krb5kdc[24758](info): commencing operation
krb5kdc: starting...
W20260928 15:06:12.201262 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:06:12 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:06:12.445478 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42017
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:12.779287 24771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:12.779579 24771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:12.779765 24771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:12.787142 24771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:06:12.787240 24771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:06:12.787318 24771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:06:12.787402 24771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:06:12.787480 24771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:12.787578 24771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:12.787660 24771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:06:12.787745 24771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:06:12.797596 24771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:12.799085 24771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:12.800452 24771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:12.812605 24777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:12.812526 24779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:12.812774 24776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:12.813866 24771 server_base.cc:1034] running on GCE node
I20260928 15:06:12.814654 24771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:12.815972 24771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:12.817239 24771 hybrid_clock.cc:648] HybridClock initialized: now 1790607972817183 us; error 36 us; skew 500 ppm
I20260928 15:06:12.817512 24771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:12 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:12.820521 24771 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:06:12.821834 24771 webserver.cc:460] Webserver started at http://127.9.12.254:40837/ using document root <none> and password file <none>
I20260928 15:06:12.822450 24771 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:12.822536 24771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:12.822768 24771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:12.825526 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "f69a79673d9141ab853c92eb841e4c6c"
format_stamp: "Formatted at 2026-09-28 15:06:12 on dist-test-slave-dmhs"
server_key: "da345c1a8fb4ab8d6641dcd9a56c3d1b"
server_key_iv: "736320dcbe006db46c36af2ed6978c7f"
server_key_version: "encryptionkey@0"
I20260928 15:06:12.826195 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "f69a79673d9141ab853c92eb841e4c6c"
format_stamp: "Formatted at 2026-09-28 15:06:12 on dist-test-slave-dmhs"
server_key: "da345c1a8fb4ab8d6641dcd9a56c3d1b"
server_key_iv: "736320dcbe006db46c36af2ed6978c7f"
server_key_version: "encryptionkey@0"
I20260928 15:06:12.831128 24771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:06:12.834640 24786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:12.835891 24771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:06:12.836103 24771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "f69a79673d9141ab853c92eb841e4c6c"
format_stamp: "Formatted at 2026-09-28 15:06:12 on dist-test-slave-dmhs"
server_key: "da345c1a8fb4ab8d6641dcd9a56c3d1b"
server_key_iv: "736320dcbe006db46c36af2ed6978c7f"
server_key_version: "encryptionkey@0"
I20260928 15:06:12.836421 24771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:12.847532 24771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:12.851476 24771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:12.851806 24771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:12.869663 24771 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:42017
I20260928 15:06:12.869683 24837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:42017 every 8 connection(s)
I20260928 15:06:12.871063 24771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:06:12.872541 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24771
I20260928 15:06:12.872766 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:06:12.873621 9267 external_mini_cluster.cc:1428] Setting key f01e7630a59e81a74c6bf6f38f461731
I20260928 15:06:12.875757 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:06:12 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:12.889416 24838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap starting.
I20260928 15:06:12.892879 24838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:12.894111 24838 log.cc:824] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:12.896966 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:12.875627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.9.12.254:42017)
0928 15:06:12.876445 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:12.876456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:12.876496 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:06:12.878020 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:12.878040 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:12.878395 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:12.878707 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:12.880503 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:12.881820 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:12.883723 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:12.884485 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:12.884605 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:12.889570 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:12.889580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:12.889597 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:12.889644 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:12.892597 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:12.893405 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:12.893411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:12.893416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:12.893561 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:12.894019 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:12.894022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:12.894023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:12.894541 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:12.894868 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:12.895288 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 15:06:12.895706 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":410,"threads_started":1}
I20260928 15:06:12.897383 24838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: No bootstrap required, opened a new log
I20260928 15:06:12.904789 24838 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:12.905146 24838 raft_consensus.cc:374] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:12.905273 24838 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69a79673d9141ab853c92eb841e4c6c, State: Initialized, Role: FOLLOWER
I20260928 15:06:12.905885 24838 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:12.906100 24838 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:06:12.906194 24838 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:06:12.906319 24838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:12.908563 24838 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:12.909205 24838 leader_election.cc:302] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69a79673d9141ab853c92eb841e4c6c; no voters:
I20260928 15:06:12.909780 24838 leader_election.cc:288] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:06:12.909921 24843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:06:12.910933 24843 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 LEADER]: Becoming Leader. State: Replica: f69a79673d9141ab853c92eb841e4c6c, State: Running, Role: LEADER
I20260928 15:06:12.911432 24843 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:12.911717 24838 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:06:12.914346 24844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f69a79673d9141ab853c92eb841e4c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } } }
I20260928 15:06:12.914752 24844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:12.915284 24845 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69a79673d9141ab853c92eb841e4c6c. Latest consensus state: current_term: 1 leader_uuid: "f69a79673d9141ab853c92eb841e4c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } } }
I20260928 15:06:12.915607 24845 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:12.920919 24850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:06:12.927682 24850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:06:12.936906 24850 catalog_manager.cc:1347] Generated new cluster ID: 381ab6adbaac4484b6746f815ce127d4
I20260928 15:06:12.937047 24850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:06:12.965103 24850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:06:12.966926 24850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:06:12.980353 24850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Generated new TSK 0
I20260928 15:06:12.981235 24850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:06:13.053722 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42017
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:13.402179 24864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:13.402463 24864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:13.402623 24864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:13.409916 24864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:13.410060 24864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:13.410249 24864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:06:13.420676 24864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:13.422020 24864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:13.423372 24864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:13.436169 24869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:13.436249 24870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:13.436201 24872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:13.441320 24864 server_base.cc:1034] running on GCE node
I20260928 15:06:13.442011 24864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:13.442991 24864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:13.444211 24864 hybrid_clock.cc:648] HybridClock initialized: now 1790607973444160 us; error 33 us; skew 500 ppm
I20260928 15:06:13.444558 24864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:13 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607973, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:13.447821 24864 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:06:13.449481 24864 webserver.cc:460] Webserver started at http://127.9.12.193:34893/ using document root <none> and password file <none>
I20260928 15:06:13.450218 24864 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:13.450351 24864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:13.450670 24864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:13.453601 24864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "a506c521b8424200a7657bd41bd1a364"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "d83beaaabaa31ff1f190497dff5edf1e"
server_key_iv: "136a111fc289640ae67d5cc1886d5645"
server_key_version: "encryptionkey@0"
I20260928 15:06:13.454347 24864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "a506c521b8424200a7657bd41bd1a364"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "d83beaaabaa31ff1f190497dff5edf1e"
server_key_iv: "136a111fc289640ae67d5cc1886d5645"
server_key_version: "encryptionkey@0"
I20260928 15:06:13.459414 24864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:06:13.462908 24879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:13.464190 24864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:06:13.464398 24864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "a506c521b8424200a7657bd41bd1a364"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "d83beaaabaa31ff1f190497dff5edf1e"
server_key_iv: "136a111fc289640ae67d5cc1886d5645"
server_key_version: "encryptionkey@0"
I20260928 15:06:13.464627 24864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:13.494829 24864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:13.498842 24864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:13.499135 24864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:13.500182 24864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:13.501519 24864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:13.501590 24864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:13.501672 24864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:13.501721 24864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:13.535032 24864 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:33889
I20260928 15:06:13.535079 24991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:33889 every 8 connection(s)
I20260928 15:06:13.536731 24864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:06:13.541370 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24864
I20260928 15:06:13.541582 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:06:13.542214 9267 external_mini_cluster.cc:1428] Setting key f211c080908935dbdbba6357d574f534
Sep 28 15:06:13 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607973, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:13.558967 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:13.539197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:50535 (local address 127.9.12.254:42017)
0928 15:06:13.539678 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:13.539684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:13.540808 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 15:06:13.543105 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:13.543109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:13.543180 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:13.543306 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:13.545200 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:13.545909 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:13.548422 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:13.548816 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:13.548921 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:13.553120 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:13.553125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:13.553128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:13.553162 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:13.555791 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:13.556635 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:13.556641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:13.556643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:13.556759 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:13.557221 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:13.557226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:13.557229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:13.557496 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:13.557656 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:13.558354 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 15:06:13.558588 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":199,"threads_started":1}
I20260928 15:06:13.559943 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:13.539830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:50535)
0928 15:06:13.540646 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:13.540697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:13.542581 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:13.543517 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:13.543525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:13.544190 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:13.545005 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:13.545034 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:13.546084 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:13.546092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:13.548248 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:13.548290 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:13.549049 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:13.549059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:13.549401 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:13.550126 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:13.550160 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:13.552873 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:13.555956 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:13.555965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:13.555981 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:13.556494 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:13.556895 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:13.556900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:13.556904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:13.557098 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:13.557626 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:13.557634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:13.557979 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 15:06:13.558237 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 15:06:13.558778 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":390,"threads_started":1}
I20260928 15:06:13.561568 24992 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42017
I20260928 15:06:13.561896 24992 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:13.562762 24992 heartbeater.cc:511] Master 127.9.12.254:42017 requested a full tablet report, sending...
I20260928 15:06:13.565060 24803 ts_manager.cc:194] Registered new tserver with Master: a506c521b8424200a7657bd41bd1a364 (127.9.12.193:33889)
I20260928 15:06:13.566454 24803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:50535
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:06:13.595402 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42017
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:13.913832 24998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:13.914127 24998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:13.914276 24998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:13.921799 24998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:13.921959 24998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:13.922132 24998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:06:13.931674 24998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:13.932991 24998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:13.934221 24998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:13.946852 25003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:13.946806 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:13.946976 25006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:13.947873 24998 server_base.cc:1034] running on GCE node
I20260928 15:06:13.948499 24998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:13.949347 24998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:13.950578 24998 hybrid_clock.cc:648] HybridClock initialized: now 1790607973950543 us; error 45 us; skew 500 ppm
I20260928 15:06:13.950870 24998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:13 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607973, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:13.953878 24998 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:06:13.955109 24998 webserver.cc:460] Webserver started at http://127.9.12.194:43939/ using document root <none> and password file <none>
I20260928 15:06:13.955724 24998 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:13.955854 24998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:13.956131 24998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:13.959009 24998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "f399666d72a04d0c942b3d55f76a65ac"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "570c923a3da2eaffe3becfac4af35461"
server_key_iv: "863e0cff77de73fb58366800306ea826"
server_key_version: "encryptionkey@0"
I20260928 15:06:13.959741 24998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "f399666d72a04d0c942b3d55f76a65ac"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "570c923a3da2eaffe3becfac4af35461"
server_key_iv: "863e0cff77de73fb58366800306ea826"
server_key_version: "encryptionkey@0"
I20260928 15:06:13.964838 24998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:06:13.968035 25013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:13.969408 24998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:13.969579 24998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "f399666d72a04d0c942b3d55f76a65ac"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "570c923a3da2eaffe3becfac4af35461"
server_key_iv: "863e0cff77de73fb58366800306ea826"
server_key_version: "encryptionkey@0"
I20260928 15:06:13.969785 24998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:13.982976 24998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:13.986599 24998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:13.986914 24998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:13.987785 24998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:13.989100 24998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:13.989197 24998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:13.989352 24998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:13.989449 24998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:14.023960 24998 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:36775
I20260928 15:06:14.024008 25125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:36775 every 8 connection(s)
I20260928 15:06:14.025389 24998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:06:14.027194 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 24998
I20260928 15:06:14.027395 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:06:14.027894 9267 external_mini_cluster.cc:1428] Setting key 7d26b8101788c0d5c994e58660d97e4b
Sep 28 15:06:14 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607973, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:14.046502 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.028111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:39293 (local address 127.9.12.254:42017)
0928 15:06:14.028354 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.028360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.029766 (+ 1406us) server_negotiation.cc:407] Connection header received
0928 15:06:14.031761 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.031770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.031865 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.031988 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:14.033471 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.034581 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.036060 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.036767 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.036915 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.040757 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:14.040763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:14.040766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:14.040797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:14.043471 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.044175 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.044180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.044182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.044357 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.044818 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.044823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.044826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.045097 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:14.045255 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.045884 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.046152 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:06:14.047446 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.028679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:39293)
0928 15:06:14.029584 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.029637 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.031503 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.032188 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.032196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.032765 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:14.033300 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.033316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.034717 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.034721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.035917 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.035928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.036970 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.036974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.037155 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.037808 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:14.037828 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:14.040519 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:14.043619 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:14.043627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:14.043642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:14.044043 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:14.044470 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:14.044475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:14.044479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:14.044700 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:14.045224 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:14.045232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:14.045546 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.045791 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.046322 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":376,"threads_started":1}
I20260928 15:06:14.048887 25126 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42017
I20260928 15:06:14.049199 25126 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:14.049917 25126 heartbeater.cc:511] Master 127.9.12.254:42017 requested a full tablet report, sending...
I20260928 15:06:14.051425 24803 ts_manager.cc:194] Registered new tserver with Master: f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
I20260928 15:06:14.052470 24803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:39293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:06:14.080745 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42017
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:14.398667 25131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:14.398936 25131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:14.399032 25131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:14.406301 25131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:14.406406 25131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:14.406519 25131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:06:14.416154 25131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:14.417315 25131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:14.418567 25131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:14.431563 25136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:14.431510 25137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:14.431449 25139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:14.432219 25131 server_base.cc:1034] running on GCE node
I20260928 15:06:14.432866 25131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:14.433786 25131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:14.435019 25131 hybrid_clock.cc:648] HybridClock initialized: now 1790607974434961 us; error 49 us; skew 500 ppm
I20260928 15:06:14.435380 25131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:14 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:14.438539 25131 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:06:14.440014 25131 webserver.cc:460] Webserver started at http://127.9.12.195:44723/ using document root <none> and password file <none>
I20260928 15:06:14.440711 25131 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:14.440807 25131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:14.441159 25131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:14.443975 25131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "13d69f5cea074d038a1f09df3d82c960"
format_stamp: "Formatted at 2026-09-28 15:06:14 on dist-test-slave-dmhs"
server_key: "5babaece8a89d3ef78b0030f0d33ac10"
server_key_iv: "f4dd79d6a0592c4f0f59334897cbb0ff"
server_key_version: "encryptionkey@0"
I20260928 15:06:14.444694 25131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "13d69f5cea074d038a1f09df3d82c960"
format_stamp: "Formatted at 2026-09-28 15:06:14 on dist-test-slave-dmhs"
server_key: "5babaece8a89d3ef78b0030f0d33ac10"
server_key_iv: "f4dd79d6a0592c4f0f59334897cbb0ff"
server_key_version: "encryptionkey@0"
I20260928 15:06:14.451584 25131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 15:06:14.454888 25146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:14.456123 25131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:06:14.456306 25131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "13d69f5cea074d038a1f09df3d82c960"
format_stamp: "Formatted at 2026-09-28 15:06:14 on dist-test-slave-dmhs"
server_key: "5babaece8a89d3ef78b0030f0d33ac10"
server_key_iv: "f4dd79d6a0592c4f0f59334897cbb0ff"
server_key_version: "encryptionkey@0"
I20260928 15:06:14.456519 25131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:14.478461 25131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:14.482894 25131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:14.483187 25131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:14.484122 25131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:14.485464 25131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:14.485580 25131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:14.485728 25131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:14.485800 25131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:14.518558 25131 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:43391
I20260928 15:06:14.518613 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:43391 every 8 connection(s)
I20260928 15:06:14.519918 25131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:06:14.523401 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25131
I20260928 15:06:14.523610 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:06:14.524150 9267 external_mini_cluster.cc:1428] Setting key 718184e4a0a3f9c5529a29252719863a
Sep 28 15:06:14 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:14.540081 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.522532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:49767 (local address 127.9.12.254:42017)
0928 15:06:14.522712 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.522716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.523997 (+ 1281us) server_negotiation.cc:407] Connection header received
0928 15:06:14.526121 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.526128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.526214 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.526295 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:14.528023 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.528816 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.530267 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.530671 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.530761 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.533974 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:14.533978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:14.533982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:14.534019 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:14.536569 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.537414 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.537421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.537425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.537616 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.538118 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.538123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.538126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.538485 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:14.538638 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.539529 (+ 891us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.539745 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:06:14.541047 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.522956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:49767)
0928 15:06:14.523825 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.523870 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.525828 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.526563 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.526574 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.527335 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:14.527874 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.527890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.528941 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.528945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.530148 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.530159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.530851 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.530856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.531053 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.531652 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:14.531677 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:14.533792 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:14.536695 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:14.536703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:14.536715 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:14.537280 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:14.537739 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:14.537745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:14.537751 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:14.538005 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:14.538707 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:14.538715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:14.539154 (+ 439us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.539450 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.539922 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":416,"threads_started":1}
I20260928 15:06:14.542227 25259 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42017
I20260928 15:06:14.542484 25259 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:14.543121 25259 heartbeater.cc:511] Master 127.9.12.254:42017 requested a full tablet report, sending...
I20260928 15:06:14.544513 24803 ts_manager.cc:194] Registered new tserver with Master: 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391)
I20260928 15:06:14.545259 24803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:49767
I20260928 15:06:14.551785 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:06:14.567433 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.555997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.9.12.254:42017)
0928 15:06:14.556150 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.556155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.556638 (+ 483us) server_negotiation.cc:407] Connection header received
0928 15:06:14.556750 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.556754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.556807 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.556886 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:14.558208 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.558893 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.560234 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.560760 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.560854 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.563129 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:14.563133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:14.563136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:14.563163 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:14.565350 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.565910 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.565913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.565915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.566028 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.566327 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.566330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.566331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.566594 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:14.566753 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.567008 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.567188 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:06:14.569631 24992 heartbeater.cc:503] Master 127.9.12.254:42017 was elected leader, sending a full tablet report...
I20260928 15:06:14.571590 24802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:06:14.578375 24802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:06:14.613268 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.590374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.9.12.194:36775)
0928 15:06:14.602427 (+ 12053us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.602441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.602507 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:06:14.602630 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.602638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.602918 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.603132 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:14.604281 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.606386 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.610371 (+ 3985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.612101 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.612256 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.612465 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.612767 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.612937 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11576,"thread_start_us":11450,"threads_started":1}
I20260928 15:06:14.613281 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.589945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36775 (local address 127.0.0.1:36126)
0928 15:06:14.601422 (+ 11477us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.601465 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.601674 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.603478 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.603489 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.603531 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:14.604037 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.604054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.606626 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.606634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.610206 (+ 3572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.610221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.612334 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.612341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.612521 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.612551 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.612674 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.612846 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11224,"mutex_wait_us":5,"thread_start_us":12656,"threads_started":1}
I20260928 15:06:14.613340 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.590043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.9.12.193:33889)
0928 15:06:14.602630 (+ 12587us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.602640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.603174 (+ 534us) server_negotiation.cc:407] Connection header received
0928 15:06:14.603280 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.603287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.603573 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.603758 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:14.604967 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.606937 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.610188 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.611636 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.611792 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.612103 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.612384 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.612573 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12392,"thread_start_us":12214,"threads_started":1}
I20260928 15:06:14.614144 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.589969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:56454)
0928 15:06:14.602829 (+ 12860us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.602843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.603006 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.604127 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.604134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.604170 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:14.604732 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.604746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.607147 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.607156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.610017 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.610042 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.611838 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.611845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.612082 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.612139 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.613550 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.613694 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12764,"mutex_wait_us":26,"thread_start_us":11043,"threads_started":1}
I20260928 15:06:14.617421 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.603013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43391 (local address 127.0.0.1:35794)
0928 15:06:14.609564 (+ 6551us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.609583 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.609691 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.611029 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.611034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.611082 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:14.611425 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.611437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.613580 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.613585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.615430 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.615442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.616649 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.616655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.616816 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.616851 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.616987 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.617119 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6399,"thread_start_us":6768,"threads_started":1}
I20260928 15:06:14.617499 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.603253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.9.12.195:43391)
0928 15:06:14.609658 (+ 6405us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.609668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.609701 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:06:14.610495 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.610503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.610719 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.610883 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:14.611631 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.613236 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.615575 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.616463 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.616605 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.616814 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.617033 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.617190 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6236,"thread_start_us":6403,"threads_started":1}
I20260928 15:06:14.617692 25061 tablet_service.cc:1467] Processing CreateTablet for tablet 05769d8dd5b54046877258b72faee90a (DEFAULT_TABLE table=test-table [id=2eb88b70be854025a72d135fa139dbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:14.617695 24927 tablet_service.cc:1467] Processing CreateTablet for tablet 05769d8dd5b54046877258b72faee90a (DEFAULT_TABLE table=test-table [id=2eb88b70be854025a72d135fa139dbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:14.619120 24927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05769d8dd5b54046877258b72faee90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:14.619120 25061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05769d8dd5b54046877258b72faee90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:14.620954 25194 tablet_service.cc:1467] Processing CreateTablet for tablet 05769d8dd5b54046877258b72faee90a (DEFAULT_TABLE table=test-table [id=2eb88b70be854025a72d135fa139dbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:14.622166 25194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05769d8dd5b54046877258b72faee90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:14.633361 25277 tablet_bootstrap.cc:492] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Bootstrap starting.
I20260928 15:06:14.635766 25279 tablet_bootstrap.cc:492] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Bootstrap starting.
I20260928 15:06:14.636518 25278 tablet_bootstrap.cc:492] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Bootstrap starting.
I20260928 15:06:14.637861 25277 tablet_bootstrap.cc:654] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:14.639165 25277 log.cc:824] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:14.639830 25279 tablet_bootstrap.cc:654] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:14.639918 25278 tablet_bootstrap.cc:654] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:14.641202 25279 log.cc:824] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:14.641260 25278 log.cc:824] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:14.642758 25277 tablet_bootstrap.cc:492] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: No bootstrap required, opened a new log
I20260928 15:06:14.643023 25277 ts_tablet_manager.cc:1397] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 15:06:14.644111 25279 tablet_bootstrap.cc:492] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: No bootstrap required, opened a new log
I20260928 15:06:14.644331 25278 tablet_bootstrap.cc:492] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: No bootstrap required, opened a new log
I20260928 15:06:14.644387 25279 ts_tablet_manager.cc:1397] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 15:06:14.644578 25278 ts_tablet_manager.cc:1397] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260928 15:06:14.650345 25277 raft_consensus.cc:348] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.650709 25277 raft_consensus.cc:374] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:14.650843 25277 raft_consensus.cc:729] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d69f5cea074d038a1f09df3d82c960, State: Initialized, Role: FOLLOWER
I20260928 15:06:14.651485 25277 consensus_queue.cc:260] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.652621 25259 heartbeater.cc:503] Master 127.9.12.254:42017 was elected leader, sending a full tablet report...
I20260928 15:06:14.653352 25277 ts_tablet_manager.cc:1428] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260928 15:06:14.654914 25279 raft_consensus.cc:348] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.654914 25278 raft_consensus.cc:348] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.655316 25279 raft_consensus.cc:374] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:14.655316 25278 raft_consensus.cc:374] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:14.655414 25278 raft_consensus.cc:729] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f399666d72a04d0c942b3d55f76a65ac, State: Initialized, Role: FOLLOWER
I20260928 15:06:14.655414 25279 raft_consensus.cc:729] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506c521b8424200a7657bd41bd1a364, State: Initialized, Role: FOLLOWER
I20260928 15:06:14.655977 25279 consensus_queue.cc:260] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.655977 25278 consensus_queue.cc:260] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.657469 25126 heartbeater.cc:503] Master 127.9.12.254:42017 was elected leader, sending a full tablet report...
I20260928 15:06:14.659924 25278 ts_tablet_manager.cc:1428] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 15:06:14.661448 25279 ts_tablet_manager.cc:1428] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260928 15:06:14.705614 25284 raft_consensus.cc:482] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:06:14.705891 25284 raft_consensus.cc:504] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.707545 25284 leader_election.cc:288] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a506c521b8424200a7657bd41bd1a364 (127.9.12.193:33889), 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391)
I20260928 15:06:14.714630 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.708058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:36463 (local address 127.9.12.195:43391)
0928 15:06:14.708535 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.708541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.708825 (+ 284us) server_negotiation.cc:407] Connection header received
0928 15:06:14.709037 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.709043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.709125 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.709206 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:14.709884 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.710645 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.712790 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.713603 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.713725 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.713776 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.714223 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.714352 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260928 15:06:14.714661 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.707993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:59341 (local address 127.9.12.193:33889)
0928 15:06:14.708344 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.708350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.709032 (+ 682us) server_negotiation.cc:407] Connection header received
0928 15:06:14.709176 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.709199 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.709278 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.709392 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:14.709993 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.710756 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.712830 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.713757 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.713925 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.714005 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.714271 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.714400 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 15:06:14.714823 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.707815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43391 (local address 127.9.12.194:36463)
0928 15:06:14.708692 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.708712 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.708890 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.709250 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.709254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.709286 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:14.709727 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.709738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.710783 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.710791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.712635 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.712647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.713819 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.713828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.714044 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.714083 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.714164 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.714349 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":84,"thread_start_us":397,"threads_started":1}
I20260928 15:06:14.715010 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.707979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.9.12.194:59341)
0928 15:06:14.708899 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:14.708917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:14.709055 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:14.709420 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:14.709424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:14.709453 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:14.709828 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.709841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.710897 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.710901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.712643 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:14.712655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.713954 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.713961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:14.714090 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.714111 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 15:06:14.714221 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:06:14.714355 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":41,"thread_start_us":585,"threads_started":1}
I20260928 15:06:14.715901 25214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05769d8dd5b54046877258b72faee90a" candidate_uuid: "f399666d72a04d0c942b3d55f76a65ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d69f5cea074d038a1f09df3d82c960" is_pre_election: true
I20260928 15:06:14.715960 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05769d8dd5b54046877258b72faee90a" candidate_uuid: "f399666d72a04d0c942b3d55f76a65ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a506c521b8424200a7657bd41bd1a364" is_pre_election: true
I20260928 15:06:14.716518 25214 raft_consensus.cc:2454] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f399666d72a04d0c942b3d55f76a65ac in term 0.
I20260928 15:06:14.716526 24947 raft_consensus.cc:2454] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f399666d72a04d0c942b3d55f76a65ac in term 0.
I20260928 15:06:14.717231 25015 leader_election.cc:302] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a506c521b8424200a7657bd41bd1a364, f399666d72a04d0c942b3d55f76a65ac; no voters:
I20260928 15:06:14.717622 25284 raft_consensus.cc:2781] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:06:14.717764 25284 raft_consensus.cc:482] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:06:14.717890 25284 raft_consensus.cc:3037] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:14.720047 25284 raft_consensus.cc:504] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.720806 25284 leader_election.cc:288] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [CANDIDATE]: Term 1 election: Requested vote from peers a506c521b8424200a7657bd41bd1a364 (127.9.12.193:33889), 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391)
I20260928 15:06:14.721124 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05769d8dd5b54046877258b72faee90a" candidate_uuid: "f399666d72a04d0c942b3d55f76a65ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a506c521b8424200a7657bd41bd1a364"
I20260928 15:06:14.721304 25214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05769d8dd5b54046877258b72faee90a" candidate_uuid: "f399666d72a04d0c942b3d55f76a65ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d69f5cea074d038a1f09df3d82c960"
I20260928 15:06:14.721374 24947 raft_consensus.cc:3037] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:14.721540 25214 raft_consensus.cc:3037] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:14.723620 24947 raft_consensus.cc:2454] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f399666d72a04d0c942b3d55f76a65ac in term 1.
I20260928 15:06:14.723817 25214 raft_consensus.cc:2454] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f399666d72a04d0c942b3d55f76a65ac in term 1.
I20260928 15:06:14.724043 25015 leader_election.cc:302] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a506c521b8424200a7657bd41bd1a364, f399666d72a04d0c942b3d55f76a65ac; no voters:
I20260928 15:06:14.724349 25284 raft_consensus.cc:2781] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:06:14.724802 25284 raft_consensus.cc:686] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 1 LEADER]: Becoming Leader. State: Replica: f399666d72a04d0c942b3d55f76a65ac, State: Running, Role: LEADER
I20260928 15:06:14.725313 25284 consensus_queue.cc:237] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } }
I20260928 15:06:14.729699 24802 catalog_manager.cc:5495] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac reported cstate change: term changed from 0 to 1, leader changed from <none> to f399666d72a04d0c942b3d55f76a65ac (127.9.12.194). New cstate: current_term: 1 leader_uuid: "f399666d72a04d0c942b3d55f76a65ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } health_report { overall_health: UNKNOWN } } }
I20260928 15:06:14.752445 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.746868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.9.12.194:36775)
0928 15:06:14.747098 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.747104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.747124 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:06:14.747325 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.747329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.747418 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.747570 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:06:14.747955 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.748710 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.750192 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.750618 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.750769 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.751275 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:06:14.751748 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:06:14.751920 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.751976 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.752055 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:06:14.760990 24947 raft_consensus.cc:1261] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Refusing update from remote peer f399666d72a04d0c942b3d55f76a65ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:14.761013 25214 raft_consensus.cc:1261] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Refusing update from remote peer f399666d72a04d0c942b3d55f76a65ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:14.762102 25284 consensus_queue.cc:1035] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:14.762459 25288 consensus_queue.cc:1035] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:06:14.771517 25260 tablet.cc:2369] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:06:14.792411 24802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:14.792734 24802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:14.796445 24802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Sending DeleteTablet for 3 replicas of tablet 05769d8dd5b54046877258b72faee90a
I20260928 15:06:14.797648 24927 tablet_service.cc:1514] Processing DeleteTablet for tablet 05769d8dd5b54046877258b72faee90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:14 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:56454
I20260928 15:06:14.797739 25194 tablet_service.cc:1514] Processing DeleteTablet for tablet 05769d8dd5b54046877258b72faee90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:14 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:35794
I20260928 15:06:14.798116 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24864
I20260928 15:06:14.798399 25061 tablet_service.cc:1514] Processing DeleteTablet for tablet 05769d8dd5b54046877258b72faee90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:14 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:36126
I20260928 15:06:14.808846 25302 tablet_replica.cc:330] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: stopping tablet replica
I20260928 15:06:14.809867 25302 raft_consensus.cc:2229] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:14.811223 25302 raft_consensus.cc:2258] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:14.813234 25304 tablet_replica.cc:330] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: stopping tablet replica
I20260928 15:06:14.813776 25304 raft_consensus.cc:2229] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:06:14.814421 25304 raft_consensus.cc:2258] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:14.814574 25302 ts_tablet_manager.cc:1905] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:14.815819 25303 tablet_replica.cc:330] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: stopping tablet replica
I20260928 15:06:14.816301 25303 raft_consensus.cc:2229] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:14.816852 25303 raft_consensus.cc:2258] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:14.819093 25304 ts_tablet_manager.cc:1905] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:14.822723 25303 ts_tablet_manager.cc:1905] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:14.827178 25304 ts_tablet_manager.cc:1918] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:14.827415 25304 log.cc:1196] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/05769d8dd5b54046877258b72faee90a
I20260928 15:06:14.827450 25302 ts_tablet_manager.cc:1918] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:14.828094 25304 ts_tablet_manager.cc:1939] T 05769d8dd5b54046877258b72faee90a P f399666d72a04d0c942b3d55f76a65ac: Deleting consensus metadata
Sep 28 15:06:14 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:06:14.829545 25302 log.cc:1196] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/05769d8dd5b54046877258b72faee90a
I20260928 15:06:14.829679 25303 ts_tablet_manager.cc:1918] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:14.829893 25303 log.cc:1196] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/05769d8dd5b54046877258b72faee90a
I20260928 15:06:14.830054 24789 catalog_manager.cc:4841] TS f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775): tablet 05769d8dd5b54046877258b72faee90a (table test-table [id=2eb88b70be854025a72d135fa139dbbb]) successfully deleted
I20260928 15:06:14.830170 25302 ts_tablet_manager.cc:1939] T 05769d8dd5b54046877258b72faee90a P a506c521b8424200a7657bd41bd1a364: Deleting consensus metadata
I20260928 15:06:14.830503 25303 ts_tablet_manager.cc:1939] T 05769d8dd5b54046877258b72faee90a P 13d69f5cea074d038a1f09df3d82c960: Deleting consensus metadata
I20260928 15:06:14.831943 24788 catalog_manager.cc:4841] TS a506c521b8424200a7657bd41bd1a364 (127.9.12.193:33889): tablet 05769d8dd5b54046877258b72faee90a (table test-table [id=2eb88b70be854025a72d135fa139dbbb]) successfully deleted
I20260928 15:06:14.832155 24789 catalog_manager.cc:4841] TS 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391): tablet 05769d8dd5b54046877258b72faee90a (table test-table [id=2eb88b70be854025a72d135fa139dbbb]) successfully deleted
I20260928 15:06:14.837438 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.799219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.9.12.193:33889)
0928 15:06:14.799383 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.799389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.815554 (+ 16165us) server_negotiation.cc:407] Connection header received
0928 15:06:14.815880 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.815886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.815969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.816062 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:14.818348 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.819500 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.822327 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.822955 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.823074 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.831682 (+ 8608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:14.831696 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:14.831714 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:14.831760 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:14.834726 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.835412 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.835418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.835421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.835539 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.835932 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.835941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.835943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.836298 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:14.836614 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:14.836866 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:06:14.837092 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:06:14.838011 24987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:14.964833 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24864
I20260928 15:06:14.984902 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24998
Sep 28 15:06:14 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:06:15.002163 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:14.985685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.9.12.194:36775)
0928 15:06:14.985879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:14.985885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:14.986006 (+ 121us) server_negotiation.cc:407] Connection header received
0928 15:06:14.986216 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:14.986221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:14.986303 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:14.986444 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:14.987704 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.988745 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.990025 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:14.990474 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:14.990605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:14.994929 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:14.994950 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:14.994965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:14.995001 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:14.998967 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:14.999726 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:14.999736 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:14.999740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:14.999898 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:15.000547 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:15.000553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:15.000556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:15.000917 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:15.001278 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:15.001567 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:06:15.001790 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:06:15.002869 25121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:15.116343 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24998
I20260928 15:06:15.135180 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 25131
Sep 28 15:06:15 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:06:15.150547 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:15.135879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.9.12.195:43391)
0928 15:06:15.136041 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:15.136045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:15.136116 (+ 71us) server_negotiation.cc:407] Connection header received
0928 15:06:15.136290 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:15.136299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:15.136372 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:15.136456 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:15.137772 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:15.138527 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:15.140032 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:15.140545 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:15.140643 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:15.145053 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:15.145067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:15.145083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:15.145117 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:15.147948 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:15.148855 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:15.148861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:15.148863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:15.148983 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:15.149304 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:15.149307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:15.149309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:15.149651 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:15.149860 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:15.150126 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:06:15.150290 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:06:15.151191 25254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:15.239156 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 25131
I20260928 15:06:15.255788 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 24771
I20260928 15:06:15.256939 24833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:15.367030 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 24771
I20260928 15:06:35.388504 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42017
--webserver_interface=127.9.12.254
--webserver_port=40837
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:35.704244 25310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:35.704555 25310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:35.704702 25310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:35.711417 25310 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:06:35.711509 25310 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:06:35.711586 25310 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:06:35.711683 25310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:06:35.711776 25310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:35.711896 25310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:35.711987 25310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:06:35.712060 25310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:06:35.721210 25310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=40837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:35.722429 25310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:35.723742 25310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:35.746135 25317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:35.746171 25319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:35.746204 25316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:35.746968 25310 server_base.cc:1034] running on GCE node
I20260928 15:06:35.747874 25310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:35.749483 25310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:35.750734 25310 hybrid_clock.cc:648] HybridClock initialized: now 1790607995750671 us; error 56 us; skew 500 ppm
I20260928 15:06:35.751133 25310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:35 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607995, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:35.760808 25310 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:06:35.762494 25310 webserver.cc:460] Webserver started at http://127.9.12.254:40837/ using document root <none> and password file <none>
I20260928 15:06:35.763334 25310 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:35.763494 25310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:35.769547 25310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:06:35.772354 25326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:35.773685 25310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:06:35.773869 25310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "f69a79673d9141ab853c92eb841e4c6c"
format_stamp: "Formatted at 2026-09-28 15:06:12 on dist-test-slave-dmhs"
server_key: "da345c1a8fb4ab8d6641dcd9a56c3d1b"
server_key_iv: "736320dcbe006db46c36af2ed6978c7f"
server_key_version: "encryptionkey@0"
I20260928 15:06:35.774984 25310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:35.795372 25310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:35.799732 25310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:35.800024 25310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:35.819201 25310 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:42017
I20260928 15:06:35.819226 25377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:42017 every 8 connection(s)
I20260928 15:06:35.820626 25310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:06:35.822543 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25310
I20260928 15:06:35.823587 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:33889
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:42017
--webserver_port=34893
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:35.830266 25378 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:06:35.834329 25378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap starting.
I20260928 15:06:35.854331 25378 log.cc:824] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:35.866115 25378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:35.866647 25378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap complete.
I20260928 15:06:35.875746 25378 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:35.876648 25378 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69a79673d9141ab853c92eb841e4c6c, State: Initialized, Role: FOLLOWER
I20260928 15:06:35.877254 25378 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:35.877513 25378 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:06:35.877629 25378 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:06:35.877831 25378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:06:35.881762 25378 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:35.882315 25378 leader_election.cc:302] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69a79673d9141ab853c92eb841e4c6c; no voters:
I20260928 15:06:35.882985 25378 leader_election.cc:288] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:06:35.883133 25382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:06:35.884366 25382 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 LEADER]: Becoming Leader. State: Replica: f69a79673d9141ab853c92eb841e4c6c, State: Running, Role: LEADER
I20260928 15:06:35.884948 25382 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:35.885277 25378 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:06:35.887436 25384 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69a79673d9141ab853c92eb841e4c6c. Latest consensus state: current_term: 2 leader_uuid: "f69a79673d9141ab853c92eb841e4c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } } }
I20260928 15:06:35.887782 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f69a79673d9141ab853c92eb841e4c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } } }
I20260928 15:06:35.888599 25384 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:35.888726 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:35.901837 25389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:06:35.909547 25389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:06:35.910959 25389 catalog_manager.cc:1259] Loaded cluster ID: 381ab6adbaac4484b6746f815ce127d4
I20260928 15:06:35.911100 25389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:06:35.915153 25389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:06:35.916909 25389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Loaded TSK: 0
I20260928 15:06:35.950213 25389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Generated new TSK 1
I20260928 15:06:35.950593 25389 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:06:36.174772 25380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:36.175169 25380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:36.175349 25380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:36.182863 25380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:36.182965 25380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:36.183172 25380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:06:36.192719 25380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:33889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=34893
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:36.193992 25380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:36.195328 25380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:36.208534 25405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:36.208812 25406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:36.209288 25408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:36.210592 25380 server_base.cc:1034] running on GCE node
I20260928 15:06:36.211222 25380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:36.212212 25380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:36.213510 25380 hybrid_clock.cc:648] HybridClock initialized: now 1790607996213454 us; error 49 us; skew 500 ppm
I20260928 15:06:36.213824 25380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:36 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:36.216951 25380 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:06:36.218534 25380 webserver.cc:460] Webserver started at http://127.9.12.193:34893/ using document root <none> and password file <none>
I20260928 15:06:36.219210 25380 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:36.219384 25380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:36.224647 25380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:36.227195 25415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:36.228534 25380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:36.228691 25380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "a506c521b8424200a7657bd41bd1a364"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "d83beaaabaa31ff1f190497dff5edf1e"
server_key_iv: "136a111fc289640ae67d5cc1886d5645"
server_key_version: "encryptionkey@0"
I20260928 15:06:36.229554 25380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:36.239486 25380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:36.243301 25380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:36.243575 25380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:36.244551 25380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:36.245910 25380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:36.246031 25380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:36.246155 25380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:36.246236 25380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:36.279786 25380 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:33889
I20260928 15:06:36.279867 25527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:33889 every 8 connection(s)
I20260928 15:06:36.281311 25380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:06:36.289077 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25380
I20260928 15:06:36.289711 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:36775
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:42017
--webserver_port=43939
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:06:36 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:36.304466 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:36.284190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:36361)
0928 15:06:36.285073 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:36.285113 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:36.286759 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:36.288018 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:36.288025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:36.288584 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:36.289125 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:36.289143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.290855 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.290860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:36.291996 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:36.292007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.293043 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.293051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:36.293344 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:36.294002 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:36.294023 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:36.296246 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:36.300575 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:36.300581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:36.300590 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:36.301008 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:36.301459 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:36.301462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:36.301464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:36.301632 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:36.302519 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:36.302524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:36.302840 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 15:06:36.303047 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 15:06:36.303431 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":327,"threads_started":1}
I20260928 15:06:36.305047 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:36.284095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:36361 (local address 127.9.12.254:42017)
0928 15:06:36.284745 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:36.284760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:36.285224 (+ 464us) server_negotiation.cc:407] Connection header received
0928 15:06:36.287262 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:36.287275 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:36.287589 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:36.287859 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:36.289359 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.290689 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.292129 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.292809 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.293002 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:36.296565 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:36.296578 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:36.296611 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:36.296680 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:36.300449 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:36.301136 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:36.301148 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:36.301157 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:36.301355 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:36.301757 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:36.301764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:36.301770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:36.302395 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:36.302623 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:36.303335 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 15:06:36.303779 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
I20260928 15:06:36.307758 25528 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42017
I20260928 15:06:36.308079 25528 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:36.308993 25528 heartbeater.cc:511] Master 127.9.12.254:42017 requested a full tablet report, sending...
I20260928 15:06:36.311585 25343 ts_manager.cc:194] Registered new tserver with Master: a506c521b8424200a7657bd41bd1a364 (127.9.12.193:33889)
I20260928 15:06:36.314016 25343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:36361
W20260928 15:06:36.609206 25532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:36.609457 25532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:36.609555 25532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:36.616355 25532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:36.616452 25532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:36.616593 25532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:06:36.625628 25532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:36775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=43939
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:36.626745 25532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:36.627977 25532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:36.641026 25538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:36.640980 25540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:36.641261 25537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:36.641925 25532 server_base.cc:1034] running on GCE node
I20260928 15:06:36.642504 25532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:36.643415 25532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:36.644631 25532 hybrid_clock.cc:648] HybridClock initialized: now 1790607996644580 us; error 63 us; skew 500 ppm
I20260928 15:06:36.644935 25532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:36 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:36.647838 25532 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:06:36.649330 25532 webserver.cc:460] Webserver started at http://127.9.12.194:43939/ using document root <none> and password file <none>
I20260928 15:06:36.649978 25532 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:36.650128 25532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:36.655128 25532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:06:36.657764 25547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:36.658947 25532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:06:36.659114 25532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "f399666d72a04d0c942b3d55f76a65ac"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "570c923a3da2eaffe3becfac4af35461"
server_key_iv: "863e0cff77de73fb58366800306ea826"
server_key_version: "encryptionkey@0"
I20260928 15:06:36.660089 25532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:36.674111 25532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:36.677875 25532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:36.678138 25532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:36.679140 25532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:36.680577 25532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:36.680663 25532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:36.680737 25532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:36.680794 25532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:36.713060 25532 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:36775
I20260928 15:06:36.713172 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:36775 every 8 connection(s)
I20260928 15:06:36.714520 25532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:06:36.715567 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25532
I20260928 15:06:36.716339 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:43391
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:42017
--webserver_port=44723
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:06:36 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:36.737294 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:36.717303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:52071 (local address 127.9.12.254:42017)
0928 15:06:36.717517 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:36.717522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:36.718727 (+ 1205us) server_negotiation.cc:407] Connection header received
0928 15:06:36.721476 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:36.721480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:36.721545 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:36.721635 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:36.723682 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.724445 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.726080 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.726905 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.727055 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:36.730325 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:36.730333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:36.730341 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:36.730385 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:36.733987 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:36.734678 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:36.734686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:36.734691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:36.734841 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:36.735227 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:36.735234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:36.735239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:36.735627 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:36.735797 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:36.736711 (+ 914us) server_negotiation.cc:299] Negotiation successful
0928 15:06:36.736953 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:06:36.738135 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:36.717759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:52071)
0928 15:06:36.718545 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:36.718598 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:36.721183 (+ 2585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:36.721898 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:36.721910 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:36.722711 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:36.723510 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:36.723537 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.724603 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.724611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:36.725919 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:36.725931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:36.727109 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:36.727114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:36.727301 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:36.727985 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:36.728007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:36.730105 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:36.734125 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:36.734131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:36.734144 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:36.734567 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:36.734959 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:36.734961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:36.734963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:36.735126 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:36.735761 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:36.735766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:36.736148 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 15:06:36.736490 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 15:06:36.736989 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
I20260928 15:06:36.739601 25660 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42017
I20260928 15:06:36.739902 25660 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:36.740653 25660 heartbeater.cc:511] Master 127.9.12.254:42017 requested a full tablet report, sending...
I20260928 15:06:36.742187 25343 ts_manager.cc:194] Registered new tserver with Master: f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
I20260928 15:06:36.743526 25343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:52071
W20260928 15:06:37.035126 25663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:37.035393 25663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:37.035495 25663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:37.042382 25663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:37.042476 25663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:37.042627 25663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:06:37.052325 25663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=44723
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:37.053542 25663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:37.055186 25663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:37.068615 25668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:37.068688 25669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:37.068650 25671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:37.069660 25663 server_base.cc:1034] running on GCE node
I20260928 15:06:37.070173 25663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:37.070947 25663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:37.072158 25663 hybrid_clock.cc:648] HybridClock initialized: now 1790607997072136 us; error 53 us; skew 500 ppm
I20260928 15:06:37.072470 25663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:37 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607997, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:37.075309 25663 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:06:37.076722 25663 webserver.cc:460] Webserver started at http://127.9.12.195:44723/ using document root <none> and password file <none>
I20260928 15:06:37.077533 25663 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:37.077677 25663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:37.083367 25663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 15:06:37.085877 25678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:37.087121 25663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:06:37.087286 25663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "13d69f5cea074d038a1f09df3d82c960"
format_stamp: "Formatted at 2026-09-28 15:06:14 on dist-test-slave-dmhs"
server_key: "5babaece8a89d3ef78b0030f0d33ac10"
server_key_iv: "f4dd79d6a0592c4f0f59334897cbb0ff"
server_key_version: "encryptionkey@0"
I20260928 15:06:37.088222 25663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:37.103525 25663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:37.107102 25663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:37.107358 25663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:37.108357 25663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:37.109534 25663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:37.109647 25663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:37.109764 25663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:37.109865 25663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:37.141398 25663 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:43391
I20260928 15:06:37.141489 25790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:43391 every 8 connection(s)
I20260928 15:06:37.142818 25663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:06:37.143355 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25663
Sep 28 15:06:37 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607997, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:06:37.158754 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.144900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.9.12.254:42017)
0928 15:06:37.145082 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.145088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.145176 (+ 88us) server_negotiation.cc:407] Connection header received
0928 15:06:37.145419 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.145425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.145502 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.145614 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:37.147264 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.148185 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.149594 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.150048 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.150178 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.153009 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:37.153015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:37.153019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:37.153060 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:37.156180 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.156846 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.156854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.156857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.156988 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.157374 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.157378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.157381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.157746 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:37.158014 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.158287 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.158456 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:06:37.165417 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.145561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:58121 (local address 127.9.12.254:42017)
0928 15:06:37.147636 (+ 2075us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.147642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.148527 (+ 885us) server_negotiation.cc:407] Connection header received
0928 15:06:37.150214 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.150219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.150300 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.150408 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:37.151871 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.152669 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.154742 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.155484 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.155615 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.159267 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:37.159272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:37.159275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:37.159302 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:37.162238 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.162940 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.162947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.162953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.163067 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.163570 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.163575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.163578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.163827 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:37.163998 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.164808 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.165024 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1970,"thread_start_us":1866,"threads_started":1}
I20260928 15:06:37.166321 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.146340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:58121)
0928 15:06:37.148383 (+ 2043us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.148428 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:37.149984 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:37.150655 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:37.150664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:37.151183 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:37.151706 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.151723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.152807 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.152812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.154603 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.154620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.155677 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.155681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.155864 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.156466 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:37.156487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:37.159036 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:37.162374 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:37.162382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:37.162396 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:37.162819 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:37.163196 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:37.163201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:37.163205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:37.163452 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:37.163978 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:37.163985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:37.164452 (+ 467us) client_negotiation.cc:769] Sending connection context
0928 15:06:37.164734 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 15:06:37.165245 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":379,"threads_started":1}
I20260928 15:06:37.167565 25791 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42017
I20260928 15:06:37.167857 25791 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:37.168609 25791 heartbeater.cc:511] Master 127.9.12.254:42017 requested a full tablet report, sending...
I20260928 15:06:37.169916 25343 ts_manager.cc:194] Registered new tserver with Master: 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391)
I20260928 15:06:37.170696 25343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:58121
I20260928 15:06:37.173629 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:06:37.179806 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.174590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.9.12.254:42017)
0928 15:06:37.174731 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.174734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.175184 (+ 450us) server_negotiation.cc:407] Connection header received
0928 15:06:37.175343 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.175346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.175481 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.175590 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:06:37.175964 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.176683 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.178172 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.178581 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.178671 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.179125 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:06:37.179368 (+ 243us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 15:06:37.179619 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33226: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 15:06:37.179929 25794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33226: token expired
W20260928 15:06:37.180598 25264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.12.254:42017)
I20260928 15:06:37.193084 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.181048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.9.12.254:42017)
0928 15:06:37.181209 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.181216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.181237 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:06:37.181418 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.181423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.181490 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.181583 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:37.182840 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.183570 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.185016 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.185476 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.185599 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.188299 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:37.188305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:37.188309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:37.188349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:37.190497 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.191190 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.191196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.191199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.191337 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.191693 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.191698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.191701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.192011 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:37.192254 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.192564 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.192769 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:06:37.195921 25343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:06:37.202867 25343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:06:37.225263 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.215530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.9.12.193:33889)
0928 15:06:37.217172 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.217179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.217200 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:06:37.217277 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.217283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.217447 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.217659 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:37.218187 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.219834 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.222553 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.223996 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.224150 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.224401 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.224757 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.224916 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1487,"thread_start_us":1384,"threads_started":1}
I20260928 15:06:37.225361 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.213840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47678)
0928 15:06:37.216069 (+ 2229us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.216090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:37.216226 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:37.217663 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:37.217670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:37.217697 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:37.218026 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.218036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.219973 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.219980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.222397 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.222411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.224244 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.224251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.224497 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.224539 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:06:37.224713 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 15:06:37.224917 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"mutex_wait_us":1,"thread_start_us":1017,"threads_started":1}
I20260928 15:06:37.228771 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.216377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36775 (local address 127.0.0.1:56178)
0928 15:06:37.216859 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.216879 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:37.216999 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:37.221817 (+ 4818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:37.221823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:37.221851 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:37.222161 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.222167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.223935 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.223941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.226204 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.226219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.227905 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.227912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.228097 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.228133 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:06:37.228230 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:06:37.228363 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":22,"thread_start_us":294,"threads_started":1}
I20260928 15:06:37.228818 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.216803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.9.12.194:36775)
0928 15:06:37.221032 (+ 4229us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.221038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.221063 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:06:37.221130 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.221135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.221320 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.221518 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:37.222314 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.223797 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.226373 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.227697 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.227840 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.228060 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.228302 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.228450 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4091,"thread_start_us":3957,"threads_started":1}
I20260928 15:06:37.229871 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.213563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43391 (local address 127.0.0.1:34946)
0928 15:06:37.215520 (+ 1957us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.215567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:37.215747 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:37.221306 (+ 5559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:37.221310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:37.221333 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:37.221752 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.221766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.223779 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.223787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.227035 (+ 3248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.227050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.229132 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.229139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.229306 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.229347 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:06:37.229427 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:06:37.229556 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":3,"thread_start_us":2418,"threads_started":1}
I20260928 15:06:37.229358 25463 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8c46a7ca3a4aff90e15eb49c188b09 (DEFAULT_TABLE table=test-table [id=14a3cf39556e4c7eb8fde3b5ce7f9c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:37.230697 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.213845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.9.12.195:43391)
0928 15:06:37.220756 (+ 6911us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.220764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.220796 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:06:37.220890 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.220899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.221120 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.221299 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:37.221933 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.223645 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.227194 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.228553 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.228676 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.230150 (+ 1474us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.230225 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.230363 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6747,"thread_start_us":6610,"threads_started":1}
I20260928 15:06:37.230815 25463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8c46a7ca3a4aff90e15eb49c188b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:37.231879 25595 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8c46a7ca3a4aff90e15eb49c188b09 (DEFAULT_TABLE table=test-table [id=14a3cf39556e4c7eb8fde3b5ce7f9c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:37.233363 25595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8c46a7ca3a4aff90e15eb49c188b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:37.234076 25726 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8c46a7ca3a4aff90e15eb49c188b09 (DEFAULT_TABLE table=test-table [id=14a3cf39556e4c7eb8fde3b5ce7f9c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:37.235343 25726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8c46a7ca3a4aff90e15eb49c188b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:37.243826 25807 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Bootstrap starting.
I20260928 15:06:37.246723 25807 tablet_bootstrap.cc:654] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:37.247756 25807 log.cc:824] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:37.249140 25809 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Bootstrap starting.
I20260928 15:06:37.250159 25810 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Bootstrap starting.
I20260928 15:06:37.250808 25807 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: No bootstrap required, opened a new log
I20260928 15:06:37.251067 25807 ts_tablet_manager.cc:1397] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 15:06:37.252866 25809 tablet_bootstrap.cc:654] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:37.253280 25810 tablet_bootstrap.cc:654] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:37.254060 25809 log.cc:824] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:37.254359 25810 log.cc:824] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:37.256968 25810 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: No bootstrap required, opened a new log
I20260928 15:06:37.257218 25810 ts_tablet_manager.cc:1397] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 15:06:37.263433 25807 raft_consensus.cc:348] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.263918 25807 raft_consensus.cc:374] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:37.264041 25807 raft_consensus.cc:729] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506c521b8424200a7657bd41bd1a364, State: Initialized, Role: FOLLOWER
I20260928 15:06:37.264109 25809 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: No bootstrap required, opened a new log
I20260928 15:06:37.264503 25809 ts_tablet_manager.cc:1397] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260928 15:06:37.264650 25807 consensus_queue.cc:260] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.265478 25810 raft_consensus.cc:348] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.265707 25528 heartbeater.cc:503] Master 127.9.12.254:42017 was elected leader, sending a full tablet report...
I20260928 15:06:37.265818 25810 raft_consensus.cc:374] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:37.265908 25810 raft_consensus.cc:729] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d69f5cea074d038a1f09df3d82c960, State: Initialized, Role: FOLLOWER
I20260928 15:06:37.266425 25810 consensus_queue.cc:260] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.267616 25807 ts_tablet_manager.cc:1428] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260928 15:06:37.268551 25791 heartbeater.cc:503] Master 127.9.12.254:42017 was elected leader, sending a full tablet report...
I20260928 15:06:37.269243 25810 ts_tablet_manager.cc:1428] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 15:06:37.274616 25809 raft_consensus.cc:348] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.274947 25809 raft_consensus.cc:374] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:37.275068 25809 raft_consensus.cc:729] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f399666d72a04d0c942b3d55f76a65ac, State: Initialized, Role: FOLLOWER
I20260928 15:06:37.275609 25809 consensus_queue.cc:260] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.276579 25660 heartbeater.cc:503] Master 127.9.12.254:42017 was elected leader, sending a full tablet report...
I20260928 15:06:37.277115 25809 ts_tablet_manager.cc:1428] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
W20260928 15:06:37.283367 25529 tablet.cc:2369] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:06:37.339413 25813 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:06:37.339720 25813 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.341440 25813 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
I20260928 15:06:37.348312 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.341881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:58177 (local address 127.9.12.195:43391)
0928 15:06:37.342049 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.342053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.342392 (+ 339us) server_negotiation.cc:407] Connection header received
0928 15:06:37.342703 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.342714 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.342794 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.342861 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:37.343799 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.344816 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.346549 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.347348 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.347519 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.347596 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.347908 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.348036 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:06:37.348503 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.342574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:36493 (local address 127.9.12.194:36775)
0928 15:06:37.342764 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.342770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.343008 (+ 238us) server_negotiation.cc:407] Connection header received
0928 15:06:37.343241 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.343249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.343331 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.343423 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:37.344195 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.345132 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.346785 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.347579 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.347753 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.347832 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.348125 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.348244 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:06:37.348748 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.341684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43391 (local address 127.9.12.193:58177)
0928 15:06:37.342297 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.342314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:37.342462 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:37.343115 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:37.343118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:37.343174 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:37.343649 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.343660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.344930 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.344938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.346408 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.346422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.347562 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.347568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.347748 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.347772 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:06:37.347881 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:06:37.348008 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":67,"thread_start_us":318,"threads_started":1}
I20260928 15:06:37.348915 25817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.342370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36775 (local address 127.9.12.193:36493)
0928 15:06:37.342882 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.342927 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:37.343067 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:37.343449 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:37.343452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:37.343478 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:37.344049 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.344061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.345253 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.345259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.346633 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:37.346643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.347800 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.347806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:37.347956 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.347980 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:06:37.348089 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:06:37.348209 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":39,"spinlock_wait_cycles":22400,"thread_start_us":402,"threads_started":1}
I20260928 15:06:37.349766 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c46a7ca3a4aff90e15eb49c188b09" candidate_uuid: "a506c521b8424200a7657bd41bd1a364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f399666d72a04d0c942b3d55f76a65ac" is_pre_election: true
I20260928 15:06:37.349766 25746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c46a7ca3a4aff90e15eb49c188b09" candidate_uuid: "a506c521b8424200a7657bd41bd1a364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d69f5cea074d038a1f09df3d82c960" is_pre_election: true
I20260928 15:06:37.350291 25615 raft_consensus.cc:2454] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a506c521b8424200a7657bd41bd1a364 in term 0.
I20260928 15:06:37.350291 25746 raft_consensus.cc:2454] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a506c521b8424200a7657bd41bd1a364 in term 0.
I20260928 15:06:37.350874 25418 leader_election.cc:302] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a506c521b8424200a7657bd41bd1a364, f399666d72a04d0c942b3d55f76a65ac; no voters:
I20260928 15:06:37.351256 25813 raft_consensus.cc:2781] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:06:37.351421 25813 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:06:37.351547 25813 raft_consensus.cc:3037] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:37.353921 25813 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.354576 25813 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 1 election: Requested vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
I20260928 15:06:37.354959 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c46a7ca3a4aff90e15eb49c188b09" candidate_uuid: "a506c521b8424200a7657bd41bd1a364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f399666d72a04d0c942b3d55f76a65ac"
I20260928 15:06:37.354959 25746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c46a7ca3a4aff90e15eb49c188b09" candidate_uuid: "a506c521b8424200a7657bd41bd1a364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d69f5cea074d038a1f09df3d82c960"
I20260928 15:06:37.355137 25615 raft_consensus.cc:3037] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:37.355137 25746 raft_consensus.cc:3037] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:37.358079 25615 raft_consensus.cc:2454] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506c521b8424200a7657bd41bd1a364 in term 1.
I20260928 15:06:37.358079 25746 raft_consensus.cc:2454] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506c521b8424200a7657bd41bd1a364 in term 1.
I20260928 15:06:37.358425 25418 leader_election.cc:302] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d69f5cea074d038a1f09df3d82c960, a506c521b8424200a7657bd41bd1a364; no voters:
I20260928 15:06:37.358669 25813 raft_consensus.cc:2781] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:06:37.359067 25813 raft_consensus.cc:686] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 LEADER]: Becoming Leader. State: Replica: a506c521b8424200a7657bd41bd1a364, State: Running, Role: LEADER
I20260928 15:06:37.359503 25813 consensus_queue.cc:237] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:37.364032 25341 catalog_manager.cc:5495] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 reported cstate change: term changed from 0 to 1, leader changed from <none> to a506c521b8424200a7657bd41bd1a364 (127.9.12.193). New cstate: current_term: 1 leader_uuid: "a506c521b8424200a7657bd41bd1a364" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } health_report { overall_health: UNKNOWN } } }
I20260928 15:06:37.386227 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.380779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.9.12.193:33889)
0928 15:06:37.380976 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.380982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.381089 (+ 107us) server_negotiation.cc:407] Connection header received
0928 15:06:37.381341 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.381345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.381427 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.381579 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:06:37.381938 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.382639 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.384043 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.384525 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.384654 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.385195 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:06:37.385669 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:06:37.385843 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.385899 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.385973 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:06:37.393924 25615 raft_consensus.cc:1261] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Refusing update from remote peer a506c521b8424200a7657bd41bd1a364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:37.393983 25746 raft_consensus.cc:1261] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Refusing update from remote peer a506c521b8424200a7657bd41bd1a364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 15:06:37.394598 25792 tablet.cc:2369] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:06:37.394850 25818 consensus_queue.cc:1035] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:37.395213 25813 consensus_queue.cc:1035] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:37.405610 25821 mvcc.cc:205] Tried to move back new op lower bound from 7334330357316411392 to 7334330357188227072. Current Snapshot: MvccSnapshot[applied={T|T < 7334330357316411392}]
I20260928 15:06:37.409777 25822 mvcc.cc:205] Tried to move back new op lower bound from 7334330357316411392 to 7334330357188227072. Current Snapshot: MvccSnapshot[applied={T|T < 7334330357316411392}]
I20260928 15:06:37.417598 25341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:37.417896 25341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:37.421427 25341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Sending DeleteTablet for 3 replicas of tablet 5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:06:37.422595 25463 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8c46a7ca3a4aff90e15eb49c188b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:37 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260928 15:06:37.422832 25726 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8c46a7ca3a4aff90e15eb49c188b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:37 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:34946
I20260928 15:06:37.422981 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 25380
I20260928 15:06:37.423542 25595 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8c46a7ca3a4aff90e15eb49c188b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:37 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:56178
I20260928 15:06:37.446488 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.423823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.9.12.193:33889)
0928 15:06:37.424488 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.424496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.424517 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:06:37.424578 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.424585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.424674 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.424789 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:37.427063 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.428634 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.431005 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.431879 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.432025 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.436534 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:37.436552 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:37.436585 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:37.436636 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:37.441424 (+ 4788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.442410 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.442422 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.442428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.442635 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.443162 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.443169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.443175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.443791 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:37.444168 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.445690 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.445961 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500}
I20260928 15:06:37.447211 25523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:37.447824 25833 tablet_replica.cc:330] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: stopping tablet replica
I20260928 15:06:37.448122 25832 tablet_replica.cc:330] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: stopping tablet replica
I20260928 15:06:37.448580 25833 raft_consensus.cc:2229] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:37.448689 25832 raft_consensus.cc:2229] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:37.449378 25833 raft_consensus.cc:2258] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:37.449380 25832 raft_consensus.cc:2258] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:37.451984 25832 ts_tablet_manager.cc:1905] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:37.452426 25833 ts_tablet_manager.cc:1905] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:37.457789 25832 ts_tablet_manager.cc:1918] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:37.457958 25832 log.cc:1196] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:06:37.458413 25832 ts_tablet_manager.cc:1939] T 5d8c46a7ca3a4aff90e15eb49c188b09 P 13d69f5cea074d038a1f09df3d82c960: Deleting consensus metadata
I20260928 15:06:37.459651 25833 ts_tablet_manager.cc:1918] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:37.459789 25833 log.cc:1196] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:06:37.459955 25329 catalog_manager.cc:4841] TS 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391): tablet 5d8c46a7ca3a4aff90e15eb49c188b09 (table test-table [id=14a3cf39556e4c7eb8fde3b5ce7f9c57]) successfully deleted
I20260928 15:06:37.460301 25833 ts_tablet_manager.cc:1939] T 5d8c46a7ca3a4aff90e15eb49c188b09 P f399666d72a04d0c942b3d55f76a65ac: Deleting consensus metadata
I20260928 15:06:37.462051 25329 catalog_manager.cc:4841] TS f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775): tablet 5d8c46a7ca3a4aff90e15eb49c188b09 (table test-table [id=14a3cf39556e4c7eb8fde3b5ce7f9c57]) successfully deleted
I20260928 15:06:37.548317 25831 tablet_replica.cc:330] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: stopping tablet replica
I20260928 15:06:37.548623 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 25380
W20260928 15:06:37.567111 25328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.12.193:33889 (error 108)
I20260928 15:06:37.567960 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 25532
I20260928 15:06:37.569175 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.568609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47686)
0928 15:06:37.568881 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.568984 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260928 15:06:37.569420 25328 catalog_manager.cc:4584] TS a506c521b8424200a7657bd41bd1a364 (127.9.12.193:33889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d8c46a7ca3a4aff90e15eb49c188b09: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
I20260928 15:06:37.581368 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.568776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.9.12.194:36775)
0928 15:06:37.568936 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.568944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.568964 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:06:37.569145 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.569151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.569245 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.569351 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:37.570788 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.571517 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.572832 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.573233 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.573325 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.575691 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:37.575704 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:37.575719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:37.575750 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:37.578767 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.579441 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.579448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.579451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.579590 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.579942 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.579945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.579947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.580379 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:37.580672 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.580967 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.581123 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:06:37.582028 25655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:37.623363 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.622802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47696)
0928 15:06:37.623079 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.623178 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 15:06:37.624470 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.624051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47704)
0928 15:06:37.624196 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.624351 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 15:06:37.668814 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 25532
I20260928 15:06:37.685024 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.684575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47706)
0928 15:06:37.684781 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.684873 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:06:37.685667 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 25663
I20260928 15:06:37.686416 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.685984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47722)
0928 15:06:37.686144 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.686251 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 15:06:37.699033 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.686430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.9.12.195:43391)
0928 15:06:37.686639 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:37.686643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:37.686660 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:06:37.686852 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:37.686857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:37.686919 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:37.687000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:37.688387 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.689103 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.690425 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:37.690871 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:37.690994 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:37.693752 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:37.693777 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:37.693792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:37.693831 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:37.696265 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.696991 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.697001 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.697004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.697170 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:37.697545 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:37.697551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:37.697554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:37.697894 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:37.698194 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:37.698501 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 15:06:37.698735 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:06:37.699635 25786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:37.766957 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.766372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47738)
0928 15:06:37.766614 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.766728 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:06:37.768093 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:37.767647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:33889 (local address 127.0.0.1:47742)
0928 15:06:37.767850 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:37.767932 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 15:06:37.786556 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 25663
I20260928 15:06:37.803437 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 25310
I20260928 15:06:37.804610 25373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:06:37.915035 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 25310
I20260928 15:06:57.936491 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42017
--webserver_interface=127.9.12.254
--webserver_port=40837
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:58.254436 25839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:58.254715 25839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:58.254859 25839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:58.262084 25839 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:06:58.262169 25839 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:06:58.262255 25839 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:06:58.262351 25839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:06:58.262421 25839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:58.262543 25839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:58.262601 25839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:06:58.262696 25839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:06:58.272197 25839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=40837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:58.273391 25839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:58.274641 25839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:58.286897 25844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:58.286988 25847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:58.286936 25845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:58.287577 25839 server_base.cc:1034] running on GCE node
I20260928 15:06:58.288369 25839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:58.289702 25839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:58.290951 25839 hybrid_clock.cc:648] HybridClock initialized: now 1790608018290879 us; error 42 us; skew 500 ppm
I20260928 15:06:58.291266 25839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:58 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:58.301263 25839 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:06:58.302628 25839 webserver.cc:460] Webserver started at http://127.9.12.254:40837/ using document root <none> and password file <none>
I20260928 15:06:58.303309 25839 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:58.303442 25839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:58.308779 25839 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:58.311371 25854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:58.312655 25839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:58.312826 25839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "f69a79673d9141ab853c92eb841e4c6c"
format_stamp: "Formatted at 2026-09-28 15:06:12 on dist-test-slave-dmhs"
server_key: "da345c1a8fb4ab8d6641dcd9a56c3d1b"
server_key_iv: "736320dcbe006db46c36af2ed6978c7f"
server_key_version: "encryptionkey@0"
I20260928 15:06:58.313736 25839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:58.326797 25839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:58.330752 25839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:58.331048 25839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:58.349412 25839 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:42017
I20260928 15:06:58.349455 25905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:42017 every 8 connection(s)
I20260928 15:06:58.350859 25839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:06:58.357636 25906 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:06:58.359479 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25839
I20260928 15:06:58.360527 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:33889
--local_ip_for_outbound_sockets=127.9.12.193
--tserver_master_addrs=127.9.12.254:42017
--webserver_port=34893
--webserver_interface=127.9.12.193
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:58.361670 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap starting.
I20260928 15:06:58.383698 25906 log.cc:824] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:58.399138 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:58.399653 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Bootstrap complete.
I20260928 15:06:58.408507 25906 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:58.409318 25906 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69a79673d9141ab853c92eb841e4c6c, State: Initialized, Role: FOLLOWER
I20260928 15:06:58.409953 25906 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:58.410190 25906 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:06:58.410305 25906 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:06:58.410490 25906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:06:58.414176 25906 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:58.414690 25906 leader_election.cc:302] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69a79673d9141ab853c92eb841e4c6c; no voters:
I20260928 15:06:58.415381 25906 leader_election.cc:288] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:06:58.415527 25910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:06:58.416898 25910 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [term 3 LEADER]: Becoming Leader. State: Replica: f69a79673d9141ab853c92eb841e4c6c, State: Running, Role: LEADER
I20260928 15:06:58.417675 25906 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:06:58.417533 25910 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } }
I20260928 15:06:58.420478 25911 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f69a79673d9141ab853c92eb841e4c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } } }
I20260928 15:06:58.420883 25911 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:58.425765 25912 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69a79673d9141ab853c92eb841e4c6c. Latest consensus state: current_term: 3 leader_uuid: "f69a79673d9141ab853c92eb841e4c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69a79673d9141ab853c92eb841e4c6c" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42017 } } }
I20260928 15:06:58.426714 25912 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:58.427989 25916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:06:58.436882 25916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:06:58.438324 25916 catalog_manager.cc:1259] Loaded cluster ID: 381ab6adbaac4484b6746f815ce127d4
I20260928 15:06:58.438473 25916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:06:58.442404 25916 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:06:58.444060 25916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Loaded TSK: 0
I20260928 15:06:58.444291 25916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f69a79673d9141ab853c92eb841e4c6c: Loaded TSK: 1
F20260928 15:06:58.445525 25916 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ace5e69ad google::LogMessage::Fail() at ??:0
@ 0x7f1ace5e988a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ace5e64de google::LogMessage::Flush() at ??:0
@ 0x7f1ace5e7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1ad234a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1ad233973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1ae40151c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1ae4014783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1ae40a02a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260928 15:06:58.486933 25929 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f1ae40a00c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1ae40ea5e9 std::function<>::operator()() at ??:0
@ 0x7f1ae4018cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1ae4016c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1ae409e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1ae409e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1ad1232345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1ad123591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1ad1235799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1ad1210eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1ad44ea6db start_thread at ??:0
@ 0x7f1acd48061f clone at ??:0
W20260928 15:06:58.759938 25908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:58.760212 25908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:58.760396 25908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:58.767714 25908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:58.767844 25908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:58.768044 25908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:06:58.778115 25908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:33889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=34893
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:58.779462 25908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:58.780841 25908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:58.793831 25953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:58.793859 25954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:58.793991 25956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:58.794616 25908 server_base.cc:1034] running on GCE node
I20260928 15:06:58.795188 25908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:58.796176 25908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:58.797432 25908 hybrid_clock.cc:648] HybridClock initialized: now 1790608018797389 us; error 42 us; skew 500 ppm
I20260928 15:06:58.797765 25908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:58 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:58.801026 25908 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:06:58.802472 25908 webserver.cc:460] Webserver started at http://127.9.12.193:34893/ using document root <none> and password file <none>
I20260928 15:06:58.803159 25908 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:58.803309 25908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:58.808704 25908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:06:58.811381 25963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:58.812667 25908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:58.812835 25908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "a506c521b8424200a7657bd41bd1a364"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "d83beaaabaa31ff1f190497dff5edf1e"
server_key_iv: "136a111fc289640ae67d5cc1886d5645"
server_key_version: "encryptionkey@0"
I20260928 15:06:58.813768 25908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:58.846648 25908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:58.850765 25908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:58.851207 25908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:58.852283 25908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:58.854231 25970 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 15:06:58.859807 25908 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:06:58.859946 25908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260928 15:06:58.860093 25908 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:06:58.863253 25908 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:06:58.863368 25908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 15:06:58.863735 25970 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Bootstrap starting.
I20260928 15:06:58.890858 25970 log.cc:824] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:58.903059 25908 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:33889
I20260928 15:06:58.903079 26077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:33889 every 8 connection(s)
I20260928 15:06:58.904665 25908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:06:58.907089 25970 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:58.907711 25970 tablet_bootstrap.cc:492] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Bootstrap complete.
I20260928 15:06:58.909067 25970 ts_tablet_manager.cc:1397] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Time spent bootstrapping tablet: real 0.046s user 0.036s sys 0.008s
I20260928 15:06:58.909634 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.907443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:46191)
0928 15:06:58.908349 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.908646 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":409,"threads_started":1}
I20260928 15:06:58.909896 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 25908
W20260928 15:06:58.910249 26078 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:42017 (0 consecutive failures): Network error: Failed to ping master at 127.9.12.254:42017: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
I20260928 15:06:58.910710 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:36775
--local_ip_for_outbound_sockets=127.9.12.194
--tserver_master_addrs=127.9.12.254:42017
--webserver_port=43939
--webserver_interface=127.9.12.194
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:58.911144 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.910745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:41195)
0928 15:06:58.910898 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.911009 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:06:58.912011 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.911638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:38079)
0928 15:06:58.911796 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.911884 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:06:58.912348 26078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:06:58.920332 25970 raft_consensus.cc:348] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:58.921285 25970 raft_consensus.cc:729] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506c521b8424200a7657bd41bd1a364, State: Initialized, Role: FOLLOWER
I20260928 15:06:58.921872 25970 consensus_queue.cc:260] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:06:58.923350 25970 ts_tablet_manager.cc:1428] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 15:06:58.923540 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.923113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:60127)
0928 15:06:58.923294 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.923398 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:06:58.924494 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.924106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:60217)
0928 15:06:58.924228 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.924368 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:06:59.235988 26081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:59.236322 26081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:59.236500 26081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:59.243759 26081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:59.243901 26081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:59.244089 26081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:06:59.253640 26081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:36775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=43939
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:59.254956 26081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:59.256228 26081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:59.269253 26087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:59.269219 26088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:59.269165 26090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:59.270033 26081 server_base.cc:1034] running on GCE node
I20260928 15:06:59.270567 26081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:59.271430 26081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:59.272651 26081 hybrid_clock.cc:648] HybridClock initialized: now 1790608019272614 us; error 52 us; skew 500 ppm
I20260928 15:06:59.272964 26081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:59 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608019, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:59.275954 26081 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:06:59.277450 26081 webserver.cc:460] Webserver started at http://127.9.12.194:43939/ using document root <none> and password file <none>
I20260928 15:06:59.278071 26081 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:59.278169 26081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:59.283228 26081 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:06:59.285774 26097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:59.287058 26081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:06:59.287187 26081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "f399666d72a04d0c942b3d55f76a65ac"
format_stamp: "Formatted at 2026-09-28 15:06:13 on dist-test-slave-dmhs"
server_key: "570c923a3da2eaffe3becfac4af35461"
server_key_iv: "863e0cff77de73fb58366800306ea826"
server_key_version: "encryptionkey@0"
I20260928 15:06:59.288077 26081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:59.322844 26081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:59.326633 26081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:59.326916 26081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:59.327893 26081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:59.329206 26081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:59.329319 26081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:59.329454 26081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:59.329541 26081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:59.362816 26081 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:36775
I20260928 15:06:59.362901 26209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:36775 every 8 connection(s)
I20260928 15:06:59.364235 26081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:06:59.367750 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26081
I20260928 15:06:59.368577 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:43391
--local_ip_for_outbound_sockets=127.9.12.195
--tserver_master_addrs=127.9.12.254:42017
--webserver_port=44723
--webserver_interface=127.9.12.195
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:59.369539 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.367178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:34025)
0928 15:06:59.368069 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.368371 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":342,"threads_started":1}
W20260928 15:06:59.370296 26210 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:42017 (0 consecutive failures): Network error: Failed to ping master at 127.9.12.254:42017: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
I20260928 15:06:59.371228 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.370801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:33095)
0928 15:06:59.370984 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.371078 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:06:59.372188 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.371747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:52121)
0928 15:06:59.371927 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.372025 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:06:59.372467 26210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:06:59.688535 26213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:59.688805 26213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:59.688997 26213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:59.696388 26213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:59.696491 26213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:59.696683 26213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:06:59.706290 26213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=44723
--tserver_master_addrs=127.9.12.254:42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:06:59.707626 26213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:59.708997 26213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:59.722328 26221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:59.722417 26219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:59.722393 26218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:59.723309 26213 server_base.cc:1034] running on GCE node
I20260928 15:06:59.723941 26213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:59.724913 26213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:59.726140 26213 hybrid_clock.cc:648] HybridClock initialized: now 1790608019726098 us; error 30 us; skew 500 ppm
I20260928 15:06:59.726449 26213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:06:59 dist-test-slave-dmhs krb5kdc[24758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608019, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:59.729554 26213 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:06:59.730885 26213 webserver.cc:460] Webserver started at http://127.9.12.195:44723/ using document root <none> and password file <none>
I20260928 15:06:59.731537 26213 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:59.731642 26213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:59.736697 26213 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:59.739151 26228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:59.740590 26213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:59.740754 26213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "13d69f5cea074d038a1f09df3d82c960"
format_stamp: "Formatted at 2026-09-28 15:06:14 on dist-test-slave-dmhs"
server_key: "5babaece8a89d3ef78b0030f0d33ac10"
server_key_iv: "f4dd79d6a0592c4f0f59334897cbb0ff"
server_key_version: "encryptionkey@0"
I20260928 15:06:59.741693 26213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:59.770138 26213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:59.774188 26213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:59.774474 26213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:59.775499 26213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:59.776904 26213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:59.776991 26213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:59.777153 26213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:59.777246 26213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:59.810139 26213 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:43391
I20260928 15:06:59.810223 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:43391 every 8 connection(s)
I20260928 15:06:59.811890 26213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:06:59.812705 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26213
I20260928 15:06:59.818255 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.816048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:56167)
0928 15:06:59.816979 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.817304 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":400,"threads_started":1}
W20260928 15:06:59.818903 26341 heartbeater.cc:644] Failed to heartbeat to 127.9.12.254:42017 (0 consecutive failures): Network error: Failed to ping master at 127.9.12.254:42017: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
I20260928 15:06:59.819645 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.819367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:58849)
0928 15:06:59.819492 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.819552 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 15:06:59.820410 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.820067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:39689)
0928 15:06:59.820176 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.820242 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 15:06:59.820632 26341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:06:59.925992 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.925257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:56305)
0928 15:06:59.925707 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.925783 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":222,"threads_started":1}
I20260928 15:07:00.371388 26346 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:00.371616 26346 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:07:00.373353 26346 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
I20260928 15:07:00.373944 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.373021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:55525)
0928 15:07:00.373541 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.373648 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
Sep 28 15:07:00 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
Sep 28 15:07:00 dist-test-slave-dmhs krb5kdc[24758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:07:00.393729 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.374043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:56067 (local address 127.9.12.195:43391)
0928 15:07:00.375209 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:00.375220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:00.375250 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:07:00.376987 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:00.376995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:00.377352 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:00.377617 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:00.379394 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.380575 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.382198 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.382806 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.382996 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:00.387154 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:00.387165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:00.387180 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:00.387220 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:00.390340 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:00.391082 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:00.391090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:00.391096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:00.391253 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:00.391641 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:00.391648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:00.391651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:00.392086 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:00.392255 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:00.392975 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 15:07:00.393315 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":892,"threads_started":1}
I20260928 15:07:00.393730 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.374800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51205 (local address 127.9.12.194:36775)
0928 15:07:00.375357 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:00.375394 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:00.375443 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:07:00.377154 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:00.377162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:00.377478 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:00.377826 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:00.379381 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.380576 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.382199 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.382803 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.383015 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:00.386726 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:00.386733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:00.386766 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:00.386824 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:00.389834 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:00.390823 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:00.390834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:00.390843 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:00.390994 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:00.391457 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:00.391460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:00.391462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:00.391990 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:00.392180 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:00.393005 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 15:07:00.393371 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
I20260928 15:07:00.393746 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.373830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:36775 (local address 127.9.12.193:51205)
0928 15:07:00.374637 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.374654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:00.374765 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:00.377806 (+ 3041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:00.377810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:00.378599 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:00.379200 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:00.379219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.380761 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.380764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:00.382064 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:00.382076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.383016 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.383022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:00.383308 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:00.384133 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:00.384150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:00.386462 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:00.389985 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:00.389992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:00.390005 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:00.390373 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:00.391126 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:00.391130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:00.391134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:00.391307 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:00.392135 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:00.392143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:00.392508 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 15:07:00.392751 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 15:07:00.393287 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":142,"spinlock_wait_cycles":8704,"thread_start_us":488,"threads_started":1}
I20260928 15:07:00.394963 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.373490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:43391 (local address 127.9.12.193:56067)
0928 15:07:00.373690 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.373727 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:00.374260 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:00.377784 (+ 3524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:00.377795 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:00.378629 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:00.379203 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:00.379219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.380770 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.380774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:00.382064 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:00.382075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:00.383064 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:00.383072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:00.383306 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:00.384067 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:00.384097 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:00.386867 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:00.390480 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:00.390486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:00.390490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:00.390942 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:00.391403 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:00.391408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:00.391411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:00.391535 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:00.392214 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:00.392218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:00.392503 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 15:07:00.392749 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 15:07:00.393434 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":379,"spinlock_wait_cycles":64896}
W20260928 15:07:00.396817 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
W20260928 15:07:00.397346 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:07:00.397498 25966 leader_election.cc:302] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a506c521b8424200a7657bd41bd1a364; no voters: 13d69f5cea074d038a1f09df3d82c960, f399666d72a04d0c942b3d55f76a65ac
I20260928 15:07:00.397892 26346 raft_consensus.cc:2726] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:07:00.821897 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.821089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:53635)
0928 15:07:00.821557 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.821676 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":227,"threads_started":1}
I20260928 15:07:00.927386 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.926758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:51237)
0928 15:07:00.927154 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.927226 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":219,"threads_started":1}
I20260928 15:07:01.376184 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:01.375525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:59437)
0928 15:07:01.375947 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:01.376025 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":234,"threads_started":1}
I20260928 15:07:01.823400 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:01.822767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:54953)
0928 15:07:01.823162 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:01.823233 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":224,"threads_started":1}
I20260928 15:07:01.929101 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:01.928223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:36127)
0928 15:07:01.928847 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:01.928917 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":309,"threads_started":1}
I20260928 15:07:02.304780 26356 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:02.304992 26356 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:07:02.305619 26356 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
W20260928 15:07:02.306641 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
W20260928 15:07:02.306900 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:07:02.307008 25966 leader_election.cc:302] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a506c521b8424200a7657bd41bd1a364; no voters: 13d69f5cea074d038a1f09df3d82c960, f399666d72a04d0c942b3d55f76a65ac
I20260928 15:07:02.307266 26356 raft_consensus.cc:2726] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:07:02.377650 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:02.377031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:33595)
0928 15:07:02.377423 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:02.377494 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":212,"threads_started":1}
I20260928 15:07:02.824826 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:02.824138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:38731)
0928 15:07:02.824581 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:02.824678 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":252,"threads_started":1}
I20260928 15:07:02.930562 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:02.929940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:37175)
0928 15:07:02.930317 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:02.930391 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":193,"threads_started":1}
I20260928 15:07:03.379177 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:03.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:48565)
0928 15:07:03.378916 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:03.379001 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260928 15:07:03.826318 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:03.825638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:41335)
0928 15:07:03.826073 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:03.826148 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260928 15:07:03.932367 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:03.931515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:54759)
0928 15:07:03.931902 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:03.931987 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":194,"threads_started":1}
I20260928 15:07:04.380818 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:04.380071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:60897)
0928 15:07:04.380593 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:04.380668 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
I20260928 15:07:04.827872 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:04.827140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:45541)
0928 15:07:04.827636 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:04.827719 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":293,"threads_started":1}
I20260928 15:07:04.933893 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:04.933182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:51951)
0928 15:07:04.933626 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:04.933713 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260928 15:07:05.382493 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.381771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:39459)
0928 15:07:05.382235 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.382311 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":266,"threads_started":1}
I20260928 15:07:05.404940 26367 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:05.405308 26367 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:07:05.405993 26367 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
W20260928 15:07:05.406994 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
W20260928 15:07:05.407192 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:07:05.407306 25966 leader_election.cc:302] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a506c521b8424200a7657bd41bd1a364; no voters: 13d69f5cea074d038a1f09df3d82c960, f399666d72a04d0c942b3d55f76a65ac
I20260928 15:07:05.407574 26367 raft_consensus.cc:2726] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:07:05.829386 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.828734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:47243)
0928 15:07:05.829154 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.829226 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":229,"threads_started":1}
I20260928 15:07:05.935323 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.934682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:43217)
0928 15:07:05.935067 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.935137 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":208,"threads_started":1}
I20260928 15:07:06.384423 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.383602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:57239)
0928 15:07:06.384120 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.384206 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":233,"threads_started":1}
I20260928 15:07:06.831018 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.830234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:59013)
0928 15:07:06.830745 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.830822 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":229,"threads_started":1}
I20260928 15:07:06.936998 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.936194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:47719)
0928 15:07:06.936753 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.936832 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":349,"threads_started":1}
I20260928 15:07:07.386229 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:07.385507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:39479)
0928 15:07:07.385955 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:07.386053 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":207,"threads_started":1}
I20260928 15:07:07.832860 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:07.832075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:60547)
0928 15:07:07.832595 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:07.832677 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":337,"threads_started":1}
I20260928 15:07:07.938745 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:07.937980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:35183)
0928 15:07:07.938410 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:07.938512 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
I20260928 15:07:08.387807 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.387115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:59643)
0928 15:07:08.387567 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.387650 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260928 15:07:08.834237 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.833633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:60139)
0928 15:07:08.834031 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.834102 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":196,"threads_started":1}
I20260928 15:07:08.940500 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.939796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:41345)
0928 15:07:08.940207 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.940329 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260928 15:07:09.389461 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.388803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:43861)
0928 15:07:09.389203 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.389278 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260928 15:07:09.836086 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.835360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:36773)
0928 15:07:09.835824 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.835914 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":247,"threads_started":1}
I20260928 15:07:09.942030 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.941338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:58427)
0928 15:07:09.941763 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.941839 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260928 15:07:10.391168 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:10.390443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:47611)
0928 15:07:10.390870 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:10.390994 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260928 15:07:10.423275 26383 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:10.423547 26383 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:07:10.424188 26383 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
W20260928 15:07:10.425173 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
W20260928 15:07:10.425415 25966 leader_election.cc:341] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775): Not found: Tablet not found: 5d8c46a7ca3a4aff90e15eb49c188b09
I20260928 15:07:10.425509 25966 leader_election.cc:302] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a506c521b8424200a7657bd41bd1a364; no voters: 13d69f5cea074d038a1f09df3d82c960, f399666d72a04d0c942b3d55f76a65ac
I20260928 15:07:10.425761 26383 raft_consensus.cc:2726] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:07:10.837558 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:10.836899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:33363)
0928 15:07:10.837306 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:10.837384 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260928 15:07:10.943506 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:10.942849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:36613)
0928 15:07:10.943257 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:10.943333 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":227,"threads_started":1}
I20260928 15:07:11.392927 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.392075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:54189)
0928 15:07:11.392653 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.392746 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":367,"threads_started":1}
I20260928 15:07:11.839120 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.838445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:37713)
0928 15:07:11.838879 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.838954 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260928 15:07:11.945286 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.944454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:34763)
0928 15:07:11.944999 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.945085 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260928 15:07:12.394500 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.393794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:36395)
0928 15:07:12.394249 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.394328 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260928 15:07:12.840910 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.840117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:35803)
0928 15:07:12.840640 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.840731 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":280,"threads_started":1}
I20260928 15:07:12.947000 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.946166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:36787)
0928 15:07:12.946678 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.946791 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":288,"threads_started":1}
I20260928 15:07:13.396252 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:13.395523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:40743)
0928 15:07:13.395977 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:13.396064 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
I20260928 15:07:13.842713 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:13.841954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:45343)
0928 15:07:13.842440 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:13.842526 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":238,"threads_started":1}
I20260928 15:07:13.949039 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:13.948269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.193:46395)
0928 15:07:13.948775 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:13.948865 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":244,"threads_started":1}
I20260928 15:07:14.398224 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:14.397424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.194:37111)
0928 15:07:14.397931 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:14.398037 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":246,"threads_started":1}
I20260928 15:07:14.823462 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 25908
I20260928 15:07:14.841151 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:14.824520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.9.12.193:33889)
0928 15:07:14.825073 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:14.825080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:14.825107 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:07:14.825170 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:14.825176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:14.825394 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:14.825531 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:14.827330 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:14.828919 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:14.830174 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:14.830718 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:14.830814 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:14.833389 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:14.833401 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:14.833421 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:14.833458 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:14.837052 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:14.837946 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:14.837955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:14.837959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:14.838113 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:14.838548 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:14.838552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:14.838556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:14.838983 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:14.839293 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:14.840524 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0928 15:07:14.840778 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260928 15:07:14.842731 26073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:14.844863 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:14.843967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42017 (local address 127.9.12.195:47569)
0928 15:07:14.844510 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:14.844649 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.254:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":328,"threads_started":1}
I20260928 15:07:14.931624 26399 raft_consensus.cc:482] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:14.931895 26399 raft_consensus.cc:504] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d69f5cea074d038a1f09df3d82c960" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 43391 } } peers { permanent_uuid: "a506c521b8424200a7657bd41bd1a364" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 33889 } } peers { permanent_uuid: "f399666d72a04d0c942b3d55f76a65ac" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 36775 } }
I20260928 15:07:14.932768 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 25908
I20260928 15:07:14.932823 26399 leader_election.cc:288] T 5d8c46a7ca3a4aff90e15eb49c188b09 P a506c521b8424200a7657bd41bd1a364 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d69f5cea074d038a1f09df3d82c960 (127.9.12.195:43391), f399666d72a04d0c942b3d55f76a65ac (127.9.12.194:36775)
W20260928 15:07:14.951589 26230 connection.cc:535] server connection from 127.9.12.193:56067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 15:07:14.951566 26100 connection.cc:535] server connection from 127.9.12.193:51205 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:07:14.951922 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26081
I20260928 15:07:14.965361 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:14.952740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.9.12.194:36775)
0928 15:07:14.953404 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:14.953409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:14.953425 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:07:14.953479 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:14.953483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:14.953561 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:14.953654 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:14.954956 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:14.955685 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:14.957019 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:14.957439 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:14.957534 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:14.960165 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:14.960170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:14.960173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:14.960203 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:14.962799 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:14.963516 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:14.963519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:14.963522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:14.963653 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:14.964004 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:14.964007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:14.964009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:14.964321 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:14.964532 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:14.964786 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 15:07:14.964971 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":361,"threads_started":1}
I20260928 15:07:14.966065 26205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:15.049431 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26081
I20260928 15:07:15.067137 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26213
I20260928 15:07:15.079926 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:15.067990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.9.12.195:43391)
0928 15:07:15.068552 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:15.068556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:15.068573 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:07:15.068627 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:15.068630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:15.068699 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:15.068787 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:15.070085 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:15.070824 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:15.071991 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:15.072463 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:15.072554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:15.075157 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:15.075167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:15.075171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:15.075200 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:15.077744 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:15.078392 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:15.078395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:15.078398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:15.078546 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:15.078831 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:15.078834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:15.078836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:15.079059 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:15.079242 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:15.079481 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 15:07:15.079645 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":315,"threads_started":1}
I20260928 15:07:15.080681 26336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:15.162431 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26213
2026-09-28T15:07:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64071 ms)
[----------] 30 tests from SecurityITest (293393 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:15 dist-test-slave-dmhs krb5kdc[26406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:15 dist-test-slave-dmhs krb5kdc[26406](info): set up 2 sockets
Sep 28 15:07:15 dist-test-slave-dmhs krb5kdc[26406](info): commencing operation
krb5kdc: starting...
W20260928 15:07:16.268837 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:16 dist-test-slave-dmhs krb5kdc[26406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:07:16.524835 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40855
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:16.853264 26419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:16.853546 26419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:16.853724 26419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:16.861136 26419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:16.861238 26419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:16.861318 26419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:16.861421 26419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:16.861492 26419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:16.870949 26419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:16.872347 26419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:16.873644 26419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:16.886394 26425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:16.886438 26427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:16.886336 26424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:16.887243 26419 server_base.cc:1034] running on GCE node
I20260928 15:07:16.888062 26419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:16.889611 26419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:16.890815 26419 hybrid_clock.cc:648] HybridClock initialized: now 1790608036890771 us; error 34 us; skew 500 ppm
I20260928 15:07:16.891141 26419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:16 dist-test-slave-dmhs krb5kdc[26406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:16.894217 26419 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:07:16.895605 26419 webserver.cc:460] Webserver started at http://127.9.12.254:45929/ using document root <none> and password file <none>
I20260928 15:07:16.896257 26419 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:16.896421 26419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:16.896682 26419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:16.899616 26419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "6b8b9f523dfd42fcaa141adacab4ed66"
format_stamp: "Formatted at 2026-09-28 15:07:16 on dist-test-slave-dmhs"
server_key: "b94c143f0dbf264eac0c313820c31a5b"
server_key_iv: "735e6ddf2dcc75eb49414f820339ce83"
server_key_version: "encryptionkey@0"
I20260928 15:07:16.900425 26419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "6b8b9f523dfd42fcaa141adacab4ed66"
format_stamp: "Formatted at 2026-09-28 15:07:16 on dist-test-slave-dmhs"
server_key: "b94c143f0dbf264eac0c313820c31a5b"
server_key_iv: "735e6ddf2dcc75eb49414f820339ce83"
server_key_version: "encryptionkey@0"
I20260928 15:07:16.905529 26419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:07:16.908859 26434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:16.910161 26419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:07:16.910346 26419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "6b8b9f523dfd42fcaa141adacab4ed66"
format_stamp: "Formatted at 2026-09-28 15:07:16 on dist-test-slave-dmhs"
server_key: "b94c143f0dbf264eac0c313820c31a5b"
server_key_iv: "735e6ddf2dcc75eb49414f820339ce83"
server_key_version: "encryptionkey@0"
I20260928 15:07:16.910537 26419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:16.954746 26419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:16.959348 26419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:16.959647 26419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:16.978026 26419 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:40855
I20260928 15:07:16.978111 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:40855 every 8 connection(s)
I20260928 15:07:16.979419 26419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:16.983340 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:16.988600 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26419
I20260928 15:07:16.988811 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:16.989464 9267 external_mini_cluster.cc:1428] Setting key 93663e1527950c6486261b120ae93071
I20260928 15:07:16.995707 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66: Bootstrap starting.
I20260928 15:07:16.999570 26486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:17.000670 26486 log.cc:824] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:17.003713 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66: No bootstrap required, opened a new log
Sep 28 15:07:17 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:17.011796 26486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40855 } }
I20260928 15:07:17.012346 26486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:17.012478 26486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8b9f523dfd42fcaa141adacab4ed66, State: Initialized, Role: FOLLOWER
I20260928 15:07:17.013139 26486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40855 } }
I20260928 15:07:17.013403 26486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:17.013541 26486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:17.013736 26486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:17.016229 26486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40855 } }
I20260928 15:07:17.016844 26486 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b8b9f523dfd42fcaa141adacab4ed66; no voters:
I20260928 15:07:17.017727 26486 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:17.017864 26491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:17.018862 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:16.991573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.9.12.254:40855)
0928 15:07:16.992524 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:16.992541 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:16.992604 (+ 63us) server_negotiation.cc:407] Connection header received
0928 15:07:16.994987 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:16.995013 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:16.995536 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:16.995981 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:16.997461 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:16.999456 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.001034 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.001874 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.002055 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:17.010246 (+ 8191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:17.010259 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:17.010280 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:17.010348 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:17.014027 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.014985 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.014995 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.015003 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.015182 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.015602 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.015607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.015611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.016140 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:17.016497 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:17.016995 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 15:07:17.017417 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":389,"threads_started":1}
I20260928 15:07:17.020394 26486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:17.020835 26491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [term 1 LEADER]: Becoming Leader. State: Replica: 6b8b9f523dfd42fcaa141adacab4ed66, State: Running, Role: LEADER
I20260928 15:07:17.021476 26491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40855 } }
I20260928 15:07:17.025331 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40855 } } }
I20260928 15:07:17.025807 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:17.025588 26493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8b9f523dfd42fcaa141adacab4ed66. Latest consensus state: current_term: 1 leader_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8b9f523dfd42fcaa141adacab4ed66" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40855 } } }
I20260928 15:07:17.026135 26493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:17.027765 26499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:17.032317 26499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:17.040872 26499 catalog_manager.cc:1347] Generated new cluster ID: 11ffae98e5cf4f66ae31c94fe1868832
I20260928 15:07:17.041007 26499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:17.059020 26499 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:17.060427 26499 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:17.067144 26499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66: Generated new TSK 0
I20260928 15:07:17.067998 26499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:07:17.133658 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40855
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:17.467283 26512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:17.467561 26512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:17.467723 26512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:17.474965 26512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:17.475093 26512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:17.475281 26512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:07:17.485059 26512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:17.486338 26512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:17.487633 26512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:17.500289 26517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:17.500352 26518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:17.500348 26520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:17.501411 26512 server_base.cc:1034] running on GCE node
I20260928 15:07:17.502044 26512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:17.502990 26512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:17.504211 26512 hybrid_clock.cc:648] HybridClock initialized: now 1790608037504164 us; error 56 us; skew 500 ppm
I20260928 15:07:17.504545 26512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:17 dist-test-slave-dmhs krb5kdc[26406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608037, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:17.507519 26512 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:07:17.508960 26512 webserver.cc:460] Webserver started at http://127.9.12.193:33111/ using document root <none> and password file <none>
I20260928 15:07:17.509632 26512 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:17.509773 26512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:17.510150 26512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:17.513131 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "bc6fa4e6be2445f58208cb58a3b83149"
format_stamp: "Formatted at 2026-09-28 15:07:17 on dist-test-slave-dmhs"
server_key: "9e87f34762dd3ed7a56b20ac82488361"
server_key_iv: "57a693dfb7cc4dfb71fce86a4a15da13"
server_key_version: "encryptionkey@0"
I20260928 15:07:17.514000 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "bc6fa4e6be2445f58208cb58a3b83149"
format_stamp: "Formatted at 2026-09-28 15:07:17 on dist-test-slave-dmhs"
server_key: "9e87f34762dd3ed7a56b20ac82488361"
server_key_iv: "57a693dfb7cc4dfb71fce86a4a15da13"
server_key_version: "encryptionkey@0"
I20260928 15:07:17.519665 26512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:07:17.523377 26527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:17.524724 26512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:17.524894 26512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "bc6fa4e6be2445f58208cb58a3b83149"
format_stamp: "Formatted at 2026-09-28 15:07:17 on dist-test-slave-dmhs"
server_key: "9e87f34762dd3ed7a56b20ac82488361"
server_key_iv: "57a693dfb7cc4dfb71fce86a4a15da13"
server_key_version: "encryptionkey@0"
I20260928 15:07:17.525097 26512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:17.539054 26512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:17.543071 26512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:17.543358 26512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:17.544382 26512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:17.545701 26512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:17.545802 26512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:17.545940 26512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:17.546012 26512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:17.579550 26512 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:41839
I20260928 15:07:17.579592 26639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:41839 every 8 connection(s)
I20260928 15:07:17.580967 26512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:17.581640 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26512
I20260928 15:07:17.581862 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:17.582753 9267 external_mini_cluster.cc:1428] Setting key b4add96d48f714fd8f410a86a862a94b
Sep 28 15:07:17 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608037, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:17.603003 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:17.583913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42023 (local address 127.9.12.254:40855)
0928 15:07:17.584615 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:17.584620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:17.585691 (+ 1071us) server_negotiation.cc:407] Connection header received
0928 15:07:17.587631 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:17.587640 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:17.587730 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:17.587852 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:17.589397 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.590605 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.592145 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.592911 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.593023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:17.597166 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:17.597171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:17.597175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:17.597207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:17.599720 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.600540 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.600544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.600546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.600670 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.601123 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.601127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.601129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.601403 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:17.601591 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:17.602362 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 15:07:17.602632 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":462,"threads_started":1}
I20260928 15:07:17.604110 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:17.584613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40855 (local address 127.9.12.193:42023)
0928 15:07:17.585519 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:17.585572 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:17.587359 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:17.588053 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:17.588061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:17.588664 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:17.589224 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:17.589241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.590761 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.590764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:17.592003 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:17.592014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.593148 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.593157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:17.593498 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:17.594244 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:17.594276 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:17.596942 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:17.599883 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:17.599891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:17.599909 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:17.600412 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:17.600780 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:17.600785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:17.600789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:17.601000 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:17.601543 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:17.601552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:17.601920 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 15:07:17.602243 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 15:07:17.602798 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":396,"threads_started":1}
I20260928 15:07:17.605746 26640 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40855
I20260928 15:07:17.606081 26640 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:17.606840 26640 heartbeater.cc:511] Master 127.9.12.254:40855 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
I20260928 15:07:17.609036 26451 ts_manager.cc:194] Registered new tserver with Master: bc6fa4e6be2445f58208cb58a3b83149 (127.9.12.193:41839)
I20260928 15:07:17.610488 26451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:42023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:07:17.643352 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40855
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:17.973003 26646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:17.973322 26646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:17.973488 26646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:17.982790 26646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:17.982969 26646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:17.983160 26646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:07:17.998600 26646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:18.000310 26646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:18.001722 26646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:18.014829 26651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:18.014860 26652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:18.014897 26654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:18.015880 26646 server_base.cc:1034] running on GCE node
I20260928 15:07:18.016543 26646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:18.017499 26646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:18.018720 26646 hybrid_clock.cc:648] HybridClock initialized: now 1790608038018670 us; error 42 us; skew 500 ppm
I20260928 15:07:18.019021 26646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:18 dist-test-slave-dmhs krb5kdc[26406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608038, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:18.022230 26646 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:07:18.023693 26646 webserver.cc:460] Webserver started at http://127.9.12.194:44261/ using document root <none> and password file <none>
I20260928 15:07:18.024420 26646 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:18.024554 26646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:18.024909 26646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:18.027774 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "02b44308dea7469292375436edfbe018"
format_stamp: "Formatted at 2026-09-28 15:07:18 on dist-test-slave-dmhs"
server_key: "52632291d969fbc174fe0732c8c53a6a"
server_key_iv: "1d0b2a74d83ea1e9c9d7d51a53e0762a"
server_key_version: "encryptionkey@0"
I20260928 15:07:18.028650 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "02b44308dea7469292375436edfbe018"
format_stamp: "Formatted at 2026-09-28 15:07:18 on dist-test-slave-dmhs"
server_key: "52632291d969fbc174fe0732c8c53a6a"
server_key_iv: "1d0b2a74d83ea1e9c9d7d51a53e0762a"
server_key_version: "encryptionkey@0"
I20260928 15:07:18.033650 26646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:07:18.036851 26661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:18.038132 26646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:18.038261 26646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "02b44308dea7469292375436edfbe018"
format_stamp: "Formatted at 2026-09-28 15:07:18 on dist-test-slave-dmhs"
server_key: "52632291d969fbc174fe0732c8c53a6a"
server_key_iv: "1d0b2a74d83ea1e9c9d7d51a53e0762a"
server_key_version: "encryptionkey@0"
I20260928 15:07:18.038427 26646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:18.052399 26646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:18.056391 26646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:18.056619 26646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:18.057526 26646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:18.058766 26646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:18.058836 26646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:18.058918 26646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:18.058971 26646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:18.091367 26646 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:44973
I20260928 15:07:18.091403 26773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:44973 every 8 connection(s)
I20260928 15:07:18.092734 26646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:18.098361 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26646
I20260928 15:07:18.098568 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:18.099174 9267 external_mini_cluster.cc:1428] Setting key 784908bbf343d1eb5ed42d18e2ef1040
Sep 28 15:07:18 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608038, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:18.114883 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.095321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:44595 (local address 127.9.12.254:40855)
0928 15:07:18.095547 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:18.095553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:18.096717 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 15:07:18.098784 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:18.098788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:18.098858 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:18.098944 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:18.100869 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.101664 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.104244 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.104716 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.104852 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.108534 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:18.108539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:18.108542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:18.108575 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:18.111479 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:18.112215 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:18.112223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:18.112229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:18.112418 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:18.112818 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:18.112826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:18.112832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:18.113229 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:18.113378 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:18.114127 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 15:07:18.114395 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:07:18.115679 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.095814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40855 (local address 127.9.12.194:44595)
0928 15:07:18.096566 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:18.096604 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:18.098484 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:18.099216 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:18.099230 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:18.099957 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:18.100703 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.100727 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.101821 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.101827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.104093 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.104113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.104939 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.104947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.105184 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.105860 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:18.105882 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:18.108304 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:18.111634 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:18.111641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:18.111655 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:18.112079 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:18.112522 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:18.112525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:18.112527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:18.112703 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:18.113352 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:18.113358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:18.113740 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 15:07:18.113965 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 15:07:18.114469 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":287,"threads_started":1}
I20260928 15:07:18.117221 26774 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40855
I20260928 15:07:18.117570 26774 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:18.118494 26774 heartbeater.cc:511] Master 127.9.12.254:40855 requested a full tablet report, sending...
I20260928 15:07:18.120170 26451 ts_manager.cc:194] Registered new tserver with Master: 02b44308dea7469292375436edfbe018 (127.9.12.194:44973)
I20260928 15:07:18.121528 26451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:44595
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:07:18.159390 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40855
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:18.503710 26779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:18.503996 26779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:18.504092 26779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:18.511294 26779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:18.511407 26779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:18.511520 26779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:07:18.521692 26779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:40855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:18.522962 26779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:18.524405 26779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:18.538156 26787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:18.538245 26784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:18.538125 26785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:18.539208 26779 server_base.cc:1034] running on GCE node
I20260928 15:07:18.539788 26779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:18.540805 26779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:18.542008 26779 hybrid_clock.cc:648] HybridClock initialized: now 1790608038541966 us; error 34 us; skew 500 ppm
I20260928 15:07:18.542327 26779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:18 dist-test-slave-dmhs krb5kdc[26406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608038, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:18.545593 26779 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:07:18.547015 26779 webserver.cc:460] Webserver started at http://127.9.12.195:43935/ using document root <none> and password file <none>
I20260928 15:07:18.547678 26779 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:18.547778 26779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:18.548034 26779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:18.550920 26779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "b99034d900614d1ca4bbce25cd9ddb92"
format_stamp: "Formatted at 2026-09-28 15:07:18 on dist-test-slave-dmhs"
server_key: "771782cb1316b9c9aeec07c7c1309be5"
server_key_iv: "8c1c7bd979b55347879025cd503589a5"
server_key_version: "encryptionkey@0"
I20260928 15:07:18.551640 26779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "b99034d900614d1ca4bbce25cd9ddb92"
format_stamp: "Formatted at 2026-09-28 15:07:18 on dist-test-slave-dmhs"
server_key: "771782cb1316b9c9aeec07c7c1309be5"
server_key_iv: "8c1c7bd979b55347879025cd503589a5"
server_key_version: "encryptionkey@0"
I20260928 15:07:18.556593 26779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 15:07:18.559787 26794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:18.561313 26779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:18.561506 26779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "b99034d900614d1ca4bbce25cd9ddb92"
format_stamp: "Formatted at 2026-09-28 15:07:18 on dist-test-slave-dmhs"
server_key: "771782cb1316b9c9aeec07c7c1309be5"
server_key_iv: "8c1c7bd979b55347879025cd503589a5"
server_key_version: "encryptionkey@0"
I20260928 15:07:18.561713 26779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:18.579232 26779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:18.584517 26779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:18.584836 26779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:18.585932 26779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:18.587261 26779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:18.587337 26779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:18.587437 26779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:18.587482 26779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:18.613826 26640 heartbeater.cc:503] Master 127.9.12.254:40855 was elected leader, sending a full tablet report...
I20260928 15:07:18.621598 26779 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:33377
I20260928 15:07:18.621732 26906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:33377 every 8 connection(s)
I20260928 15:07:18.623042 26779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:18.625571 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26779
I20260928 15:07:18.625758 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:18.626322 9267 external_mini_cluster.cc:1428] Setting key 5d3da8e1393c93e384c62dedeb1ab1cf
Sep 28 15:07:18 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608038, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:18.644613 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.625739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:42373 (local address 127.9.12.254:40855)
0928 15:07:18.626339 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:18.626343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:18.627553 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 15:07:18.629831 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:18.629836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:18.629927 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:18.630044 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:18.631535 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.632678 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.634237 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.634642 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.634776 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.638302 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:18.638307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:18.638310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:18.638345 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:18.641458 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:18.642337 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:18.642341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:18.642343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:18.642503 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:18.642927 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:18.642930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:18.642933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:18.643218 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:18.643335 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:18.644078 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 15:07:18.644250 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
I20260928 15:07:18.645627 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.626448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:40855 (local address 127.9.12.195:42373)
0928 15:07:18.627403 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:18.627447 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:18.629578 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:18.630295 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:18.630305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:18.630820 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:18.631367 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.631384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.632843 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.632848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.634093 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.634105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.634869 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.634876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.635116 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.635814 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:18.635844 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:18.638092 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:18.641662 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:18.641681 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:18.641697 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:18.642201 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:18.642617 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:18.642621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:18.642623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:18.642842 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:18.643363 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:18.643369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:18.643707 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 15:07:18.643992 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 15:07:18.644473 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":491,"threads_started":1}
I20260928 15:07:18.647171 26907 heartbeater.cc:347] Connected to a master server at 127.9.12.254:40855
I20260928 15:07:18.647503 26907 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:18.648341 26907 heartbeater.cc:511] Master 127.9.12.254:40855 requested a full tablet report, sending...
I20260928 15:07:18.649780 26451 ts_manager.cc:194] Registered new tserver with Master: b99034d900614d1ca4bbce25cd9ddb92 (127.9.12.195:33377)
I20260928 15:07:18.650630 26451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:42373
I20260928 15:07:18.654737 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:18.671806 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.659296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.9.12.254:40855)
0928 15:07:18.659526 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:18.659535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:18.659971 (+ 436us) server_negotiation.cc:407] Connection header received
0928 15:07:18.660106 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:18.660110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:18.660175 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:18.660260 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:18.661975 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.662680 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.663987 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.664433 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.664557 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.666987 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:18.666992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:18.666995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:18.667024 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:18.669474 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:18.670105 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:18.670108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:18.670110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:18.670226 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:18.670547 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:18.670550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:18.670552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:18.670848 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:18.671035 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:18.671326 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 15:07:18.671495 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:07:18.675805 26451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:18.682816 26451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:18.730180 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.695215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:44973 (local address 127.0.0.1:46440)
0928 15:07:18.718973 (+ 23758us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:18.719048 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:18.719307 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:18.721922 (+ 2615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:18.721939 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:18.721973 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:18.722367 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.722380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.724230 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.724236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.727018 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.727035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.728994 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.729004 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.729234 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.729319 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 15:07:18.729467 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 15:07:18.729667 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23412,"mutex_wait_us":47,"thread_start_us":25555,"threads_started":1}
I20260928 15:07:18.730400 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.695687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.9.12.194:44973)
0928 15:07:18.720596 (+ 24909us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:18.720606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:18.720644 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:07:18.721045 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:18.721070 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:18.721377 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:18.721582 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:18.722655 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.724081 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.727230 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.728495 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.728643 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.728874 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:18.729874 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0928 15:07:18.730055 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24739,"thread_start_us":24727,"threads_started":1}
I20260928 15:07:18.730454 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.696305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.9.12.195:33377)
0928 15:07:18.718443 (+ 22138us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:18.718456 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:18.721166 (+ 2710us) server_negotiation.cc:407] Connection header received
0928 15:07:18.721468 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:18.721497 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:18.721729 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:18.721905 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:18.722919 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.724586 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.727205 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.728658 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.728794 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.729053 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:18.729969 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 15:07:18.730133 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21906,"thread_start_us":21734,"threads_started":1}
I20260928 15:07:18.731014 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.695712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:33377 (local address 127.0.0.1:47784)
0928 15:07:18.721041 (+ 25329us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:18.721059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:18.721263 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:18.722273 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:18.722277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:18.722299 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:18.722750 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.722759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.724763 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.724768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.727029 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.727037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.729213 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.729219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.729410 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.729441 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:07:18.729527 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:07:18.729664 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25202,"mutex_wait_us":47,"thread_start_us":25101,"threads_started":1}
I20260928 15:07:18.732149 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.695309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:41839 (local address 127.0.0.1:52444)
0928 15:07:18.719982 (+ 24673us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:18.720001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:18.720109 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:18.723982 (+ 3873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:18.723989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:18.724022 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:18.724544 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.724558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.726307 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.726314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.728673 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:18.728687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.731356 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.731361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:18.731550 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.731580 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:07:18.731657 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 15:07:18.731786 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24543,"mutex_wait_us":23,"thread_start_us":24382,"threads_started":1}
I20260928 15:07:18.732453 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:18.695733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.9.12.193:41839)
0928 15:07:18.723414 (+ 27681us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:18.723423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:18.723462 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:07:18.723554 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:18.723563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:18.723769 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:18.723954 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:18.724799 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.726171 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.728851 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:18.730867 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:18.731011 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:18.731218 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:18.731911 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 15:07:18.732087 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27489,"thread_start_us":27324,"threads_started":1}
I20260928 15:07:18.734943 26709 tablet_service.cc:1467] Processing CreateTablet for tablet b0e77c42175c45eba75e365b901b6af6 (DEFAULT_TABLE table=test-table [id=493b794b5a6c4ba3ab3bcc65e0585eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:18.735569 26842 tablet_service.cc:1467] Processing CreateTablet for tablet b0e77c42175c45eba75e365b901b6af6 (DEFAULT_TABLE table=test-table [id=493b794b5a6c4ba3ab3bcc65e0585eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:18.735983 26575 tablet_service.cc:1467] Processing CreateTablet for tablet b0e77c42175c45eba75e365b901b6af6 (DEFAULT_TABLE table=test-table [id=493b794b5a6c4ba3ab3bcc65e0585eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:18.736557 26709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e77c42175c45eba75e365b901b6af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:18.736976 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e77c42175c45eba75e365b901b6af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:18.737398 26575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e77c42175c45eba75e365b901b6af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:18.748493 26926 tablet_bootstrap.cc:492] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Bootstrap starting.
I20260928 15:07:18.751561 26928 tablet_bootstrap.cc:492] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Bootstrap starting.
I20260928 15:07:18.752197 26926 tablet_bootstrap.cc:654] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:18.752425 26927 tablet_bootstrap.cc:492] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Bootstrap starting.
I20260928 15:07:18.753793 26926 log.cc:824] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:18.755980 26927 tablet_bootstrap.cc:654] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:18.756068 26928 tablet_bootstrap.cc:654] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:18.757189 26928 log.cc:824] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:18.757313 26927 log.cc:824] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:18.757679 26926 tablet_bootstrap.cc:492] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: No bootstrap required, opened a new log
I20260928 15:07:18.758008 26926 ts_tablet_manager.cc:1397] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 15:07:18.760092 26927 tablet_bootstrap.cc:492] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: No bootstrap required, opened a new log
I20260928 15:07:18.760424 26928 tablet_bootstrap.cc:492] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: No bootstrap required, opened a new log
I20260928 15:07:18.760432 26927 ts_tablet_manager.cc:1397] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260928 15:07:18.760680 26928 ts_tablet_manager.cc:1397] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260928 15:07:18.768625 26927 raft_consensus.cc:348] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:18.769044 26927 raft_consensus.cc:374] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:18.768800 26926 raft_consensus.cc:348] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:18.769140 26927 raft_consensus.cc:729] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6fa4e6be2445f58208cb58a3b83149, State: Initialized, Role: FOLLOWER
I20260928 15:07:18.769235 26926 raft_consensus.cc:374] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:18.769345 26926 raft_consensus.cc:729] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99034d900614d1ca4bbce25cd9ddb92, State: Initialized, Role: FOLLOWER
I20260928 15:07:18.769934 26927 consensus_queue.cc:260] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:18.769946 26926 consensus_queue.cc:260] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:18.771574 26927 ts_tablet_manager.cc:1428] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 15:07:18.772755 26928 raft_consensus.cc:348] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:18.773198 26928 raft_consensus.cc:374] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:18.773240 26907 heartbeater.cc:503] Master 127.9.12.254:40855 was elected leader, sending a full tablet report...
I20260928 15:07:18.773293 26928 raft_consensus.cc:729] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b44308dea7469292375436edfbe018, State: Initialized, Role: FOLLOWER
I20260928 15:07:18.774026 26926 ts_tablet_manager.cc:1428] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260928 15:07:18.773877 26928 consensus_queue.cc:260] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:18.775434 26774 heartbeater.cc:503] Master 127.9.12.254:40855 was elected leader, sending a full tablet report...
I20260928 15:07:18.776086 26928 ts_tablet_manager.cc:1428] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260928 15:07:18.833814 26641 tablet.cc:2369] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:18.844702 26775 tablet.cc:2369] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:18.874805 26908 tablet.cc:2369] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:19.061959 26934 raft_consensus.cc:482] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:19.062335 26934 raft_consensus.cc:504] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:19.064018 26934 leader_election.cc:288] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc6fa4e6be2445f58208cb58a3b83149 (127.9.12.193:41839), b99034d900614d1ca4bbce25cd9ddb92 (127.9.12.195:33377)
I20260928 15:07:19.071065 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.064497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:39637 (local address 127.9.12.193:41839)
0928 15:07:19.064745 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.064753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.065589 (+ 836us) server_negotiation.cc:407] Connection header received
0928 15:07:19.065807 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.065830 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.065916 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.066036 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:19.066649 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.067379 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.069261 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.070052 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.070241 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.070337 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.070664 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.070792 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:07:19.071084 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.064684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:45729 (local address 127.9.12.195:33377)
0928 15:07:19.064921 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.064929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.065504 (+ 575us) server_negotiation.cc:407] Connection header received
0928 15:07:19.065746 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.065750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.065809 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.065906 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:19.066499 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.067232 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.069263 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.070050 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.070255 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.070351 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.070678 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.070810 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:07:19.071201 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.064473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:33377 (local address 127.9.12.194:45729)
0928 15:07:19.065384 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:19.065400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:19.065584 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:19.065944 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:19.065948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:19.065972 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:19.066345 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:19.066354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.067366 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.067374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:19.069107 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:19.069119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.070342 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.070348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:19.070524 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.070546 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:07:19.070649 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:07:19.070768 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":110,"thread_start_us":540,"threads_started":1}
I20260928 15:07:19.071861 26936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.064480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:41839 (local address 127.9.12.194:39637)
0928 15:07:19.065483 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:19.065497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:19.065645 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:19.066053 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:19.066057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:19.066095 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:19.066503 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:19.066511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.067544 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.067549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:19.069112 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:19.069124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.070306 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.070312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:19.070499 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.070532 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:07:19.070637 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:07:19.070766 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":97,"thread_start_us":590,"threads_started":1}
I20260928 15:07:19.072365 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e77c42175c45eba75e365b901b6af6" candidate_uuid: "02b44308dea7469292375436edfbe018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b99034d900614d1ca4bbce25cd9ddb92" is_pre_election: true
I20260928 15:07:19.072901 26862 raft_consensus.cc:2454] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b44308dea7469292375436edfbe018 in term 0.
I20260928 15:07:19.072961 26595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e77c42175c45eba75e365b901b6af6" candidate_uuid: "02b44308dea7469292375436edfbe018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6fa4e6be2445f58208cb58a3b83149" is_pre_election: true
I20260928 15:07:19.073506 26595 raft_consensus.cc:2454] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b44308dea7469292375436edfbe018 in term 0.
I20260928 15:07:19.073570 26662 leader_election.cc:302] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b44308dea7469292375436edfbe018, b99034d900614d1ca4bbce25cd9ddb92; no voters:
I20260928 15:07:19.074028 26934 raft_consensus.cc:2781] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:19.074182 26934 raft_consensus.cc:482] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:19.074288 26934 raft_consensus.cc:3037] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:19.076508 26934 raft_consensus.cc:504] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:19.077104 26934 leader_election.cc:288] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [CANDIDATE]: Term 1 election: Requested vote from peers bc6fa4e6be2445f58208cb58a3b83149 (127.9.12.193:41839), b99034d900614d1ca4bbce25cd9ddb92 (127.9.12.195:33377)
I20260928 15:07:19.077431 26595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e77c42175c45eba75e365b901b6af6" candidate_uuid: "02b44308dea7469292375436edfbe018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6fa4e6be2445f58208cb58a3b83149"
I20260928 15:07:19.077633 26595 raft_consensus.cc:3037] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:19.077610 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e77c42175c45eba75e365b901b6af6" candidate_uuid: "02b44308dea7469292375436edfbe018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b99034d900614d1ca4bbce25cd9ddb92"
I20260928 15:07:19.077803 26862 raft_consensus.cc:3037] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:19.079823 26595 raft_consensus.cc:2454] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b44308dea7469292375436edfbe018 in term 1.
I20260928 15:07:19.079980 26862 raft_consensus.cc:2454] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b44308dea7469292375436edfbe018 in term 1.
I20260928 15:07:19.080219 26665 leader_election.cc:302] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b44308dea7469292375436edfbe018, bc6fa4e6be2445f58208cb58a3b83149; no voters:
I20260928 15:07:19.080559 26934 raft_consensus.cc:2781] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:19.081007 26934 raft_consensus.cc:686] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 1 LEADER]: Becoming Leader. State: Replica: 02b44308dea7469292375436edfbe018, State: Running, Role: LEADER
I20260928 15:07:19.081475 26934 consensus_queue.cc:237] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } }
I20260928 15:07:19.086220 26449 catalog_manager.cc:5495] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02b44308dea7469292375436edfbe018 (127.9.12.194). New cstate: current_term: 1 leader_uuid: "02b44308dea7469292375436edfbe018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b44308dea7469292375436edfbe018" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 } health_report { overall_health: UNKNOWN } } }
I20260928 15:07:19.103955 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.097984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.9.12.194:44973)
0928 15:07:19.098149 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.098153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.098169 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:07:19.098316 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.098320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.098440 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.098560 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:19.099009 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.099755 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.101793 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.102199 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.102288 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.102865 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:19.103333 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:19.103445 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.103587 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.103653 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:07:19.111917 26595 raft_consensus.cc:1261] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 1 FOLLOWER]: Refusing update from remote peer 02b44308dea7469292375436edfbe018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:19.111917 26862 raft_consensus.cc:1261] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 1 FOLLOWER]: Refusing update from remote peer 02b44308dea7469292375436edfbe018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:19.112859 26937 consensus_queue.cc:1035] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b99034d900614d1ca4bbce25cd9ddb92" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:19.113242 26934 consensus_queue.cc:1035] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6fa4e6be2445f58208cb58a3b83149" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:19.124598 26940 mvcc.cc:205] Tried to move back new op lower bound from 7334330528193064960 to 7334330528081657856. Current Snapshot: MvccSnapshot[applied={T|T < 7334330528193064960}]
I20260928 15:07:19.138546 26450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:19.138872 26450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:19.142943 26450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b8b9f523dfd42fcaa141adacab4ed66: Sending DeleteTablet for 3 replicas of tablet b0e77c42175c45eba75e365b901b6af6
I20260928 15:07:19.144094 26575 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e77c42175c45eba75e365b901b6af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:19 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:52444
I20260928 15:07:19.144080 26709 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e77c42175c45eba75e365b901b6af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:19 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:46440
I20260928 15:07:19.145048 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e77c42175c45eba75e365b901b6af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:19 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:47784
I20260928 15:07:19.145186 26950 tablet_replica.cc:330] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: stopping tablet replica
I20260928 15:07:19.145862 26950 raft_consensus.cc:2229] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:19.147109 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26512
I20260928 15:07:19.147094 26950 raft_consensus.cc:2258] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:19.148206 26949 tablet_replica.cc:330] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: stopping tablet replica
I20260928 15:07:19.148761 26949 raft_consensus.cc:2229] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:19.149701 26949 raft_consensus.cc:2258] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:19.152195 26949 ts_tablet_manager.cc:1905] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:19.152370 26950 ts_tablet_manager.cc:1905] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:19.160492 26951 tablet_replica.cc:330] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: stopping tablet replica
I20260928 15:07:19.160933 26951 raft_consensus.cc:2229] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:19.161470 26951 raft_consensus.cc:2258] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:19.161609 26949 ts_tablet_manager.cc:1918] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:19.161811 26949 log.cc:1196] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/b0e77c42175c45eba75e365b901b6af6
I20260928 15:07:19.162604 26949 ts_tablet_manager.cc:1939] T b0e77c42175c45eba75e365b901b6af6 P bc6fa4e6be2445f58208cb58a3b83149: Deleting consensus metadata
I20260928 15:07:19.163045 26950 ts_tablet_manager.cc:1918] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:19.163213 26950 log.cc:1196] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/b0e77c42175c45eba75e365b901b6af6
I20260928 15:07:19.163800 26950 ts_tablet_manager.cc:1939] T b0e77c42175c45eba75e365b901b6af6 P 02b44308dea7469292375436edfbe018: Deleting consensus metadata
I20260928 15:07:19.164990 26438 catalog_manager.cc:4841] TS bc6fa4e6be2445f58208cb58a3b83149 (127.9.12.193:41839): tablet b0e77c42175c45eba75e365b901b6af6 (table test-table [id=493b794b5a6c4ba3ab3bcc65e0585eb2]) successfully deleted
I20260928 15:07:19.165622 26436 catalog_manager.cc:4841] TS 02b44308dea7469292375436edfbe018 (127.9.12.194:44973): tablet b0e77c42175c45eba75e365b901b6af6 (table test-table [id=493b794b5a6c4ba3ab3bcc65e0585eb2]) successfully deleted
I20260928 15:07:19.167119 26951 ts_tablet_manager.cc:1905] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:07:19 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:07:19.173772 26951 ts_tablet_manager.cc:1918] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:19.173967 26951 log.cc:1196] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/b0e77c42175c45eba75e365b901b6af6
I20260928 15:07:19.174553 26951 ts_tablet_manager.cc:1939] T b0e77c42175c45eba75e365b901b6af6 P b99034d900614d1ca4bbce25cd9ddb92: Deleting consensus metadata
I20260928 15:07:19.176589 26435 catalog_manager.cc:4841] TS b99034d900614d1ca4bbce25cd9ddb92 (127.9.12.195:33377): tablet b0e77c42175c45eba75e365b901b6af6 (table test-table [id=493b794b5a6c4ba3ab3bcc65e0585eb2]) successfully deleted
I20260928 15:07:19.180188 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.149746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.9.12.193:41839)
0928 15:07:19.149931 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.149937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.160310 (+ 10373us) server_negotiation.cc:407] Connection header received
0928 15:07:19.160800 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.160806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.160892 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.160977 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:19.162899 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.164068 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.166870 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.167526 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.167668 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.173239 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:19.173257 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:19.173276 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:19.173320 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:19.177207 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:19.178129 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:19.178136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:19.178138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:19.178263 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:19.178701 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:19.178704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:19.178706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:19.179071 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:19.179294 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.179696 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.179869 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:07:19.180946 26635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:19.272742 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26512
I20260928 15:07:19.291065 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26646
Sep 28 15:07:19 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:07:19.308586 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.291963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.9.12.194:44973)
0928 15:07:19.292146 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.292153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.292223 (+ 70us) server_negotiation.cc:407] Connection header received
0928 15:07:19.292490 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.292496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.292584 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.292693 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:19.294037 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.294851 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.296296 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.296756 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.296894 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.301463 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:19.301485 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:19.301506 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:19.301557 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:19.305593 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:19.306336 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:19.306345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:19.306349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:19.306532 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:19.306889 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:19.306894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:19.306898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:19.307358 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:19.307679 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.307892 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.308132 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:07:19.309377 26769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:19.423413 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26646
I20260928 15:07:19.442550 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26779
Sep 28 15:07:19 dist-test-slave-dmhs krb5kdc[26406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:07:19.458060 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.443382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.9.12.195:33377)
0928 15:07:19.443552 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.443556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.443667 (+ 111us) server_negotiation.cc:407] Connection header received
0928 15:07:19.443846 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.443850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.443916 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.444011 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:19.445486 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.446220 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.447442 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.447822 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.447913 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.452592 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:19.452608 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:19.452625 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:19.452684 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:19.455306 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:19.456029 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:19.456036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:19.456038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:19.456178 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:19.456689 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:19.456692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:19.456694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:19.457061 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:19.457294 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.457607 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.457773 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:07:19.458752 26902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:19.549006 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26779
I20260928 15:07:19.565689 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26419
I20260928 15:07:19.566885 26481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:19.679176 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26419
2026-09-28T15:07:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4519 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T15:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:19Z Disabled control of system clock
I20260928 15:07:19.757571 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33943
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:20.086848 26960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:20.087157 26960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:20.087245 26960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:20.094578 26960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:20.094689 26960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:20.094734 26960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:20.094791 26960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:20.094835 26960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:20.103766 26960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:33943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:20.105125 26960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:20.106864 26960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:20.118543 26966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:20.118594 26965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:20.118558 26968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:20.119549 26960 server_base.cc:1034] running on GCE node
I20260928 15:07:20.120471 26960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:20.121915 26960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:20.123157 26960 hybrid_clock.cc:648] HybridClock initialized: now 1790608040123108 us; error 41 us; skew 500 ppm
I20260928 15:07:20.123507 26960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:20.125918 26960 webserver.cc:460] Webserver started at http://127.9.12.254:34731/ using document root <none> and password file <none>
I20260928 15:07:20.126631 26960 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:20.126760 26960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:20.127091 26960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:20.130079 26960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "9319c6b02d844ca8b050fa4d41a36d7e"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-dmhs"
server_key: "980b4dd924d035ccc63095ec9aa33651"
server_key_iv: "1bde74ba744ecc25153e206af786780a"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.130896 26960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "9319c6b02d844ca8b050fa4d41a36d7e"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-dmhs"
server_key: "980b4dd924d035ccc63095ec9aa33651"
server_key_iv: "1bde74ba744ecc25153e206af786780a"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.135910 26960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:07:20.139330 26974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:20.140753 26960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 15:07:20.140934 26960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "9319c6b02d844ca8b050fa4d41a36d7e"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-dmhs"
server_key: "980b4dd924d035ccc63095ec9aa33651"
server_key_iv: "1bde74ba744ecc25153e206af786780a"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.141145 26960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:20.178957 26960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:20.179911 26960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:20.180233 26960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:20.199090 26960 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:33943
I20260928 15:07:20.199110 27025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:33943 every 8 connection(s)
I20260928 15:07:20.200709 26960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:20.201664 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 26960
I20260928 15:07:20.201850 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:20.202407 9267 external_mini_cluster.cc:1428] Setting key b22167f30efa1fe6ec1abfc6b0891c7b
I20260928 15:07:20.205467 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:20.216629 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:20.204469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.9.12.254:33943)
0928 15:07:20.205507 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:20.205541 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:20.205598 (+ 57us) server_negotiation.cc:407] Connection header received
0928 15:07:20.207295 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:20.207316 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:20.207753 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:20.208098 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:20.208783 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:20.210104 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:20.212068 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:20.212779 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:20.212910 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:20.214247 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:20.214254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:20.214265 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:20.214829 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:20.214909 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:20.214931 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:20.215030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:20.215254 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 15:07:20.215603 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":383,"threads_started":1}
I20260928 15:07:20.217898 27026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e: Bootstrap starting.
I20260928 15:07:20.220960 27026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:20.221933 27026 log.cc:824] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:20.224562 27026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e: No bootstrap required, opened a new log
I20260928 15:07:20.233012 27026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33943 } }
I20260928 15:07:20.233371 27026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:20.233465 27026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9319c6b02d844ca8b050fa4d41a36d7e, State: Initialized, Role: FOLLOWER
I20260928 15:07:20.234017 27026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33943 } }
I20260928 15:07:20.234239 27026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:20.234370 27026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:20.234565 27026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:20.236739 27026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33943 } }
I20260928 15:07:20.237264 27026 leader_election.cc:302] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9319c6b02d844ca8b050fa4d41a36d7e; no voters:
I20260928 15:07:20.237815 27026 leader_election.cc:288] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:20.237970 27031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:20.238745 27031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [term 1 LEADER]: Becoming Leader. State: Replica: 9319c6b02d844ca8b050fa4d41a36d7e, State: Running, Role: LEADER
I20260928 15:07:20.239275 27031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33943 } }
I20260928 15:07:20.240063 27026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:20.242601 27033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9319c6b02d844ca8b050fa4d41a36d7e. Latest consensus state: current_term: 1 leader_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33943 } } }
I20260928 15:07:20.242914 27032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9319c6b02d844ca8b050fa4d41a36d7e" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 33943 } } }
I20260928 15:07:20.243193 27032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:20.243165 27033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:20.247681 27040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:20.253036 27040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:20.261296 27040 catalog_manager.cc:1347] Generated new cluster ID: 727c85c2b0ac4e68878d52eec72438f0
I20260928 15:07:20.261417 27040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:20.297585 27040 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:20.298939 27040 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:20.305917 27040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e: Generated new TSK 0
I20260928 15:07:20.306772 27040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:20.313050 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33943
--builtin_ntp_servers=127.9.12.212:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:20.664788 27050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:20.665066 27050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:20.665227 27050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:20.672665 27050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:20.672768 27050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:20.672951 27050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:07:20.681975 27050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:20.683297 27050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:20.685115 27050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:20.698061 27058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:20.698122 27056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:20.698160 27055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:20.698926 27050 server_base.cc:1034] running on GCE node
I20260928 15:07:20.699472 27050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:20.700395 27050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:20.701607 27050 hybrid_clock.cc:648] HybridClock initialized: now 1790608040701553 us; error 47 us; skew 500 ppm
I20260928 15:07:20.701962 27050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:20.704151 27050 webserver.cc:460] Webserver started at http://127.9.12.193:39081/ using document root <none> and password file <none>
I20260928 15:07:20.704865 27050 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:20.704988 27050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:20.705312 27050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:20.708114 27050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "4bfc5def331241088ebc6d5610e94b15"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-dmhs"
server_key: "8123d8222d0b6c0a73bffd4a96404158"
server_key_iv: "62f7b716a003a9fa2bb3e30c57d5dc89"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.708973 27050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "4bfc5def331241088ebc6d5610e94b15"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-dmhs"
server_key: "8123d8222d0b6c0a73bffd4a96404158"
server_key_iv: "62f7b716a003a9fa2bb3e30c57d5dc89"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.714200 27050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:07:20.717756 27064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:20.719122 27050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:07:20.719252 27050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "4bfc5def331241088ebc6d5610e94b15"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-dmhs"
server_key: "8123d8222d0b6c0a73bffd4a96404158"
server_key_iv: "62f7b716a003a9fa2bb3e30c57d5dc89"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.719440 27050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:20.735069 27050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:20.735863 27050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:20.736121 27050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:20.737147 27050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:20.738541 27050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:20.738624 27050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:20.738708 27050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:20.738761 27050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:20.773027 27050 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:37793
I20260928 15:07:20.773082 27176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:37793 every 8 connection(s)
I20260928 15:07:20.774374 27050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:20.782310 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27050
I20260928 15:07:20.782498 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:20.782972 9267 external_mini_cluster.cc:1428] Setting key ab09f208072146205995d760bc6a6b72
I20260928 15:07:20.785696 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33943
--builtin_ntp_servers=127.9.12.212:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:20.787425 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:20.777167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:59277 (local address 127.9.12.254:33943)
0928 15:07:20.777668 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:20.777672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:20.778714 (+ 1042us) server_negotiation.cc:407] Connection header received
0928 15:07:20.780643 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:20.780649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:20.780745 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:20.780863 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:20.782021 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:20.782851 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:20.784536 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:20.784969 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:20.785100 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:20.786539 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:20.786545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:20.786549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:20.786574 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:20.786613 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:20.786625 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:20.786720 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:20.786979 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:07:20.787081 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260928 15:07:20.788614 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:20.777651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33943 (local address 127.9.12.193:59277)
0928 15:07:20.778545 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:20.778582 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:20.780350 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:20.781119 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:20.781128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:20.781222 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:20.781821 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:20.781838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:20.783042 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:20.783049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:20.784386 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:20.784404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:20.785200 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:20.785207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:20.785445 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:20.786216 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:20.786219 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:20.786261 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:20.786338 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:20.786718 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:20.786730 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:20.786740 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:07:20.786955 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 15:07:20.787369 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":288,"threads_started":1}
I20260928 15:07:20.790241 27177 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33943
I20260928 15:07:20.790537 27177 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:20.791292 27177 heartbeater.cc:511] Master 127.9.12.254:33943 requested a full tablet report, sending...
I20260928 15:07:20.793601 26991 ts_manager.cc:194] Registered new tserver with Master: 4bfc5def331241088ebc6d5610e94b15 (127.9.12.193:37793)
I20260928 15:07:20.795186 26991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.193:59277
W20260928 15:07:21.111814 27181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:21.112144 27181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:21.112354 27181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:21.119728 27181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:21.119884 27181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:21.120066 27181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:07:21.129312 27181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:21.130661 27181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:21.132467 27181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:21.146026 27187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:21.146111 27186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:21.146134 27189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:21.149784 27181 server_base.cc:1034] running on GCE node
I20260928 15:07:21.150421 27181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:21.151458 27181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:21.152688 27181 hybrid_clock.cc:648] HybridClock initialized: now 1790608041152623 us; error 61 us; skew 500 ppm
I20260928 15:07:21.153007 27181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:21.155313 27181 webserver.cc:460] Webserver started at http://127.9.12.194:45423/ using document root <none> and password file <none>
I20260928 15:07:21.156075 27181 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:21.156204 27181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:21.156554 27181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:21.159391 27181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "0c76176663fc4a32b5c8fd0c38742fe7"
format_stamp: "Formatted at 2026-09-28 15:07:21 on dist-test-slave-dmhs"
server_key: "2a2d867200f84f75e94b91d850488dad"
server_key_iv: "e0612bded1a352930fe7bdd928f1ca57"
server_key_version: "encryptionkey@0"
I20260928 15:07:21.160181 27181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "0c76176663fc4a32b5c8fd0c38742fe7"
format_stamp: "Formatted at 2026-09-28 15:07:21 on dist-test-slave-dmhs"
server_key: "2a2d867200f84f75e94b91d850488dad"
server_key_iv: "e0612bded1a352930fe7bdd928f1ca57"
server_key_version: "encryptionkey@0"
I20260928 15:07:21.165241 27181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:07:21.168655 27195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:21.169884 27181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:07:21.170059 27181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "0c76176663fc4a32b5c8fd0c38742fe7"
format_stamp: "Formatted at 2026-09-28 15:07:21 on dist-test-slave-dmhs"
server_key: "2a2d867200f84f75e94b91d850488dad"
server_key_iv: "e0612bded1a352930fe7bdd928f1ca57"
server_key_version: "encryptionkey@0"
I20260928 15:07:21.170284 27181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:21.183089 27181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:21.183919 27181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:21.184223 27181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:21.185277 27181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:21.186638 27181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:21.186753 27181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:21.186847 27181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:21.186911 27181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:21.220921 27181 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:45411
I20260928 15:07:21.221038 27307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:45411 every 8 connection(s)
I20260928 15:07:21.222219 27181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:21.222822 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27181
I20260928 15:07:21.223008 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:21.223609 9267 external_mini_cluster.cc:1428] Setting key 0007ac582ad2655fc361bbf27a62a787
I20260928 15:07:21.226984 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33943
--builtin_ntp_servers=127.9.12.212:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:21.236016 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.225096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:54675 (local address 127.9.12.254:33943)
0928 15:07:21.225277 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.225283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.226663 (+ 1380us) server_negotiation.cc:407] Connection header received
0928 15:07:21.228586 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.228591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.228683 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.228794 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:21.229840 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.230570 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.232526 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.233256 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.233400 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.235101 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:21.235111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:21.235117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:21.235141 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:21.235187 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:21.235204 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:21.235308 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.235540 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.235654 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:07:21.236871 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.225580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33943 (local address 127.9.12.194:54675)
0928 15:07:21.226513 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.226551 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.228329 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.229031 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.229039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.229136 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:21.229674 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.229690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.230722 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.230734 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.232366 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.232382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.233561 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.233571 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.233867 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.234687 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:21.234691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:21.234783 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:21.234902 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:21.235318 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:21.235327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:21.235333 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.235498 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.235798 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":350,"threads_started":1}
I20260928 15:07:21.238296 27308 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33943
I20260928 15:07:21.238626 27308 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:21.239378 27308 heartbeater.cc:511] Master 127.9.12.254:33943 requested a full tablet report, sending...
I20260928 15:07:21.240815 26991 ts_manager.cc:194] Registered new tserver with Master: 0c76176663fc4a32b5c8fd0c38742fe7 (127.9.12.194:45411)
I20260928 15:07:21.241742 26991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.194:54675
W20260928 15:07:21.556588 27311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:21.556860 27311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:21.557010 27311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:21.564209 27311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:21.564389 27311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:21.564572 27311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:07:21.573920 27311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:21.575300 27311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:21.577054 27311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:21.590301 27316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:21.590464 27319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:21.590583 27317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:21.591660 27311 server_base.cc:1034] running on GCE node
I20260928 15:07:21.592196 27311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:21.593206 27311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:21.594416 27311 hybrid_clock.cc:648] HybridClock initialized: now 1790608041594376 us; error 35 us; skew 500 ppm
I20260928 15:07:21.594691 27311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:21.596872 27311 webserver.cc:460] Webserver started at http://127.9.12.195:38983/ using document root <none> and password file <none>
I20260928 15:07:21.597553 27311 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:21.597640 27311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:21.597896 27311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:21.600699 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "f46cc96154d34338a47d81deefed9371"
format_stamp: "Formatted at 2026-09-28 15:07:21 on dist-test-slave-dmhs"
server_key: "62eb724013af71d6f0836ab5c13bf3b4"
server_key_iv: "ed36fed3f643cfd22407de59bbf14d8f"
server_key_version: "encryptionkey@0"
I20260928 15:07:21.601444 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "f46cc96154d34338a47d81deefed9371"
format_stamp: "Formatted at 2026-09-28 15:07:21 on dist-test-slave-dmhs"
server_key: "62eb724013af71d6f0836ab5c13bf3b4"
server_key_iv: "ed36fed3f643cfd22407de59bbf14d8f"
server_key_version: "encryptionkey@0"
I20260928 15:07:21.606457 27311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 15:07:21.609925 27325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:21.611300 27311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:07:21.611430 27311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "f46cc96154d34338a47d81deefed9371"
format_stamp: "Formatted at 2026-09-28 15:07:21 on dist-test-slave-dmhs"
server_key: "62eb724013af71d6f0836ab5c13bf3b4"
server_key_iv: "ed36fed3f643cfd22407de59bbf14d8f"
server_key_version: "encryptionkey@0"
I20260928 15:07:21.611626 27311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:21.624382 27311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:21.625185 27311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:21.625447 27311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:21.626358 27311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:21.627751 27311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:21.627836 27311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:21.627908 27311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:21.627962 27311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:21.661744 27311 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:34319
I20260928 15:07:21.661793 27437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:34319 every 8 connection(s)
I20260928 15:07:21.663115 27311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:21.664566 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27311
I20260928 15:07:21.664758 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:21.665333 9267 external_mini_cluster.cc:1428] Setting key 48c1586a39855bfcdaa9409feb11d99e
I20260928 15:07:21.676587 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.665997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:40863 (local address 127.9.12.254:33943)
0928 15:07:21.666201 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.666207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.668371 (+ 2164us) server_negotiation.cc:407] Connection header received
0928 15:07:21.670253 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.670258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.670332 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.670415 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:21.671442 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.672325 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.673958 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.674395 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.674527 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.675774 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:21.675779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:21.675782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:21.675799 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:21.675830 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:21.675842 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:21.675911 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.676101 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.676178 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:07:21.677606 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.666777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:33943 (local address 127.9.12.195:40863)
0928 15:07:21.668189 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.668237 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.669951 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.670623 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.670631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.670716 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:21.671286 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.671302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.672476 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.672480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.673808 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.673820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.674606 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.674611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.674815 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.675495 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:21.675497 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:21.675535 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:21.675607 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:21.675925 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:21.675933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:21.675938 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.676071 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.676462 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":625,"threads_started":1}
I20260928 15:07:21.678931 27438 heartbeater.cc:347] Connected to a master server at 127.9.12.254:33943
I20260928 15:07:21.679265 27438 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:21.680104 27438 heartbeater.cc:511] Master 127.9.12.254:33943 requested a full tablet report, sending...
I20260928 15:07:21.681488 26991 ts_manager.cc:194] Registered new tserver with Master: f46cc96154d34338a47d81deefed9371 (127.9.12.195:34319)
I20260928 15:07:21.682274 26991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.195:40863
I20260928 15:07:21.693105 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:21.702816 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.697174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.9.12.254:33943)
0928 15:07:21.697362 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.697366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.697783 (+ 417us) server_negotiation.cc:407] Connection header received
0928 15:07:21.697919 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.697924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.698001 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.698081 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:21.698546 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.699228 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.700654 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.701048 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.701143 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.702264 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:21.702268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:21.702272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:21.702288 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:21.702318 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:21.702330 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:21.702398 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.702526 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.702597 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:07:21.706604 26991 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:21.713388 26991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:21.736976 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.726384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.9.12.193:37793)
0928 15:07:21.728968 (+ 2584us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.728981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.729004 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:07:21.729086 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.729092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.729264 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.729478 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:21.730204 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.731770 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.734288 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.735667 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.735802 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.736066 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.736457 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.736644 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2443,"thread_start_us":2345,"threads_started":1}
I20260928 15:07:21.736986 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.725914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37793 (local address 127.0.0.1:46454)
0928 15:07:21.726872 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.726916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.727108 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.729472 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.729479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.729506 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:21.729973 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.730003 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.731933 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.731937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.734095 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.734107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.735865 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.735870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.736187 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.736231 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.736396 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.736584 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":52,"thread_start_us":3890,"threads_started":1}
I20260928 15:07:21.738754 27454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.726186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:34319 (local address 127.0.0.1:54430)
0928 15:07:21.727872 (+ 1686us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.727888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.728006 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.731355 (+ 3349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.731359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.731389 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:21.731889 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.731903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.733457 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.733465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.736127 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.736142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.737931 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.737937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.738113 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.738153 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.738250 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.738375 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"mutex_wait_us":189,"thread_start_us":294,"threads_started":1}
I20260928 15:07:21.739554 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.727293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:45411 (local address 127.0.0.1:39422)
0928 15:07:21.730137 (+ 2844us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.730153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.730250 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.730837 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.730844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.730870 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:21.731630 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.731643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.733687 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.733692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.736127 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.736142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.738605 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.738611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.738990 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.739027 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.739111 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.739228 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2702,"mutex_wait_us":113,"thread_start_us":850,"threads_started":1}
I20260928 15:07:21.740216 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.727927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.9.12.194:45411)
0928 15:07:21.728493 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.728502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.730228 (+ 1726us) server_negotiation.cc:407] Connection header received
0928 15:07:21.730389 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.730400 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.730641 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.730831 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:21.731856 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.733543 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.736669 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.738388 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.738572 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.738784 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.739808 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.739979 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":343,"threads_started":1}
I20260928 15:07:21.741427 27112 tablet_service.cc:1467] Processing CreateTablet for tablet 5d44cd1cdb1e4958a3cafc6bbb494412 (DEFAULT_TABLE table=test-table [id=33e20b83cc584ae5a10366ac6d194d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:21.742967 27112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d44cd1cdb1e4958a3cafc6bbb494412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:21.743898 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.726391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.9.12.195:34319)
0928 15:07:21.730791 (+ 4400us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.730799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.730829 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:07:21.730922 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.730929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.731138 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.732068 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:21.732332 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.733323 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.736395 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.737732 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.737879 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.738164 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.738290 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.743559 (+ 5269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4220,"thread_start_us":4073,"threads_started":1}
I20260928 15:07:21.743727 27243 tablet_service.cc:1467] Processing CreateTablet for tablet 5d44cd1cdb1e4958a3cafc6bbb494412 (DEFAULT_TABLE table=test-table [id=33e20b83cc584ae5a10366ac6d194d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:21.745136 27243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d44cd1cdb1e4958a3cafc6bbb494412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:21.747418 27373 tablet_service.cc:1467] Processing CreateTablet for tablet 5d44cd1cdb1e4958a3cafc6bbb494412 (DEFAULT_TABLE table=test-table [id=33e20b83cc584ae5a10366ac6d194d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:21.748658 27373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d44cd1cdb1e4958a3cafc6bbb494412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:21.760200 27456 tablet_bootstrap.cc:492] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Bootstrap starting.
I20260928 15:07:21.760808 27458 tablet_bootstrap.cc:492] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Bootstrap starting.
I20260928 15:07:21.763880 27458 tablet_bootstrap.cc:654] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:21.764891 27456 tablet_bootstrap.cc:654] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:21.765133 27457 tablet_bootstrap.cc:492] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Bootstrap starting.
I20260928 15:07:21.765455 27458 log.cc:824] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:21.766181 27456 log.cc:824] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:21.769517 27458 tablet_bootstrap.cc:492] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: No bootstrap required, opened a new log
I20260928 15:07:21.769812 27458 ts_tablet_manager.cc:1397] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260928 15:07:21.769999 27457 tablet_bootstrap.cc:654] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:21.771268 27457 log.cc:824] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:21.781085 27458 raft_consensus.cc:348] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.782573 27458 raft_consensus.cc:374] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:21.782757 27458 raft_consensus.cc:729] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c76176663fc4a32b5c8fd0c38742fe7, State: Initialized, Role: FOLLOWER
I20260928 15:07:21.782811 27457 tablet_bootstrap.cc:492] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: No bootstrap required, opened a new log
I20260928 15:07:21.783166 27457 ts_tablet_manager.cc:1397] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20260928 15:07:21.784403 27456 tablet_bootstrap.cc:492] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: No bootstrap required, opened a new log
I20260928 15:07:21.783639 27458 consensus_queue.cc:260] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.784739 27456 ts_tablet_manager.cc:1397] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.007s
I20260928 15:07:21.791796 27308 heartbeater.cc:503] Master 127.9.12.254:33943 was elected leader, sending a full tablet report...
I20260928 15:07:21.792833 27458 ts_tablet_manager.cc:1428] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Time spent starting tablet: real 0.023s user 0.013s sys 0.007s
I20260928 15:07:21.795496 27457 raft_consensus.cc:348] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.795499 27456 raft_consensus.cc:348] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.795967 27457 raft_consensus.cc:374] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:21.795967 27456 raft_consensus.cc:374] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:21.796078 27456 raft_consensus.cc:729] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfc5def331241088ebc6d5610e94b15, State: Initialized, Role: FOLLOWER
I20260928 15:07:21.796078 27457 raft_consensus.cc:729] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46cc96154d34338a47d81deefed9371, State: Initialized, Role: FOLLOWER
I20260928 15:07:21.796744 27457 consensus_queue.cc:260] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.796744 27456 consensus_queue.cc:260] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.797931 27177 heartbeater.cc:503] Master 127.9.12.254:33943 was elected leader, sending a full tablet report...
I20260928 15:07:21.798904 27456 ts_tablet_manager.cc:1428] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 15:07:21.799922 27438 heartbeater.cc:503] Master 127.9.12.254:33943 was elected leader, sending a full tablet report...
I20260928 15:07:21.800521 27457 ts_tablet_manager.cc:1428] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260928 15:07:21.888440 27464 raft_consensus.cc:482] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:21.888839 27464 raft_consensus.cc:504] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.890627 27464 leader_election.cc:288] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bfc5def331241088ebc6d5610e94b15 (127.9.12.193:37793), 0c76176663fc4a32b5c8fd0c38742fe7 (127.9.12.194:45411)
I20260928 15:07:21.898003 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.891210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:36221 (local address 127.9.12.193:37793)
0928 15:07:21.891414 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.891421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.891525 (+ 104us) server_negotiation.cc:407] Connection header received
0928 15:07:21.891767 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.891773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.891888 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.891968 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:21.892488 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.893475 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.896001 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.897198 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.897373 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.897447 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.897635 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.897759 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:07:21.898063 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.891054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:37793 (local address 127.9.12.195:36221)
0928 15:07:21.891399 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.891430 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.891571 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.892066 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.892071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.892094 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:21.892323 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.892333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.893630 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.893634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.895802 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.895817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.897351 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.897355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.897473 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.897496 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.897567 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.897723 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":26}
I20260928 15:07:21.898443 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.891212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:50739 (local address 127.9.12.194:45411)
0928 15:07:21.891440 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.891449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.892050 (+ 601us) server_negotiation.cc:407] Connection header received
0928 15:07:21.892197 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.892201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.892325 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.892427 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:21.892914 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.893827 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.896247 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.897521 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.897686 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.897750 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.898045 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.898206 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:07:21.898468 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.891108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:45411 (local address 127.9.12.195:50739)
0928 15:07:21.891906 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:21.891919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:21.892066 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:21.892457 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:21.892463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:21.892491 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:21.892766 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.892772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.893994 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.893999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.896085 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:21.896095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.897704 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.897709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:21.897871 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.897904 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:07:21.897990 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:07:21.898114 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":31,"thread_start_us":469,"threads_started":1}
I20260928 15:07:21.899425 27263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d44cd1cdb1e4958a3cafc6bbb494412" candidate_uuid: "f46cc96154d34338a47d81deefed9371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" is_pre_election: true
I20260928 15:07:21.899423 27132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d44cd1cdb1e4958a3cafc6bbb494412" candidate_uuid: "f46cc96154d34338a47d81deefed9371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfc5def331241088ebc6d5610e94b15" is_pre_election: true
I20260928 15:07:21.900017 27132 raft_consensus.cc:2454] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46cc96154d34338a47d81deefed9371 in term 0.
I20260928 15:07:21.900015 27263 raft_consensus.cc:2454] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46cc96154d34338a47d81deefed9371 in term 0.
I20260928 15:07:21.900744 27329 leader_election.cc:302] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c76176663fc4a32b5c8fd0c38742fe7, f46cc96154d34338a47d81deefed9371; no voters:
I20260928 15:07:21.901147 27464 raft_consensus.cc:2781] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:21.901288 27464 raft_consensus.cc:482] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:21.901387 27464 raft_consensus.cc:3037] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:21.903566 27464 raft_consensus.cc:504] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
I20260928 15:07:21.904212 27464 leader_election.cc:288] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [CANDIDATE]: Term 1 election: Requested vote from peers 4bfc5def331241088ebc6d5610e94b15 (127.9.12.193:37793), 0c76176663fc4a32b5c8fd0c38742fe7 (127.9.12.194:45411)
I20260928 15:07:21.904646 27132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d44cd1cdb1e4958a3cafc6bbb494412" candidate_uuid: "f46cc96154d34338a47d81deefed9371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfc5def331241088ebc6d5610e94b15"
I20260928 15:07:21.904745 27263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d44cd1cdb1e4958a3cafc6bbb494412" candidate_uuid: "f46cc96154d34338a47d81deefed9371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c76176663fc4a32b5c8fd0c38742fe7"
I20260928 15:07:21.904870 27132 raft_consensus.cc:3037] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:21.904979 27263 raft_consensus.cc:3037] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:21.908207 27132 raft_consensus.cc:2454] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46cc96154d34338a47d81deefed9371 in term 1.
I20260928 15:07:21.908207 27263 raft_consensus.cc:2454] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46cc96154d34338a47d81deefed9371 in term 1.
I20260928 15:07:21.908705 27328 leader_election.cc:302] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfc5def331241088ebc6d5610e94b15, f46cc96154d34338a47d81deefed9371; no voters:
I20260928 15:07:21.909075 27464 raft_consensus.cc:2781] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:21.909524 27464 raft_consensus.cc:686] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 1 LEADER]: Becoming Leader. State: Replica: f46cc96154d34338a47d81deefed9371, State: Running, Role: LEADER
I20260928 15:07:21.910014 27464 consensus_queue.cc:237] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } }
W20260928 15:07:21.914938 27439 tablet.cc:2369] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:21.914948 26990 catalog_manager.cc:5495] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 reported cstate change: term changed from 0 to 1, leader changed from <none> to f46cc96154d34338a47d81deefed9371 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "f46cc96154d34338a47d81deefed9371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46cc96154d34338a47d81deefed9371" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 } health_report { overall_health: UNKNOWN } } }
I20260928 15:07:21.932199 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.926640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.9.12.195:34319)
0928 15:07:21.926848 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.926854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.926987 (+ 133us) server_negotiation.cc:407] Connection header received
0928 15:07:21.927228 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.927232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.927315 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.927466 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:21.927829 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.928640 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.930069 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.930495 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.930631 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.931144 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:21.931615 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:21.931779 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.931860 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.931946 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:07:21.940469 27263 raft_consensus.cc:1261] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 1 FOLLOWER]: Refusing update from remote peer f46cc96154d34338a47d81deefed9371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:21.940472 27132 raft_consensus.cc:1261] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 1 FOLLOWER]: Refusing update from remote peer f46cc96154d34338a47d81deefed9371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:21.941459 27464 consensus_queue.cc:1035] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bfc5def331241088ebc6d5610e94b15" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:21.941740 27466 consensus_queue.cc:1035] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c76176663fc4a32b5c8fd0c38742fe7" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 45411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:21.956857 27470 mvcc.cc:205] Tried to move back new op lower bound from 7334330539777556480 to 7334330539667447808. Current Snapshot: MvccSnapshot[applied={T|T < 7334330539777556480}]
I20260928 15:07:21.965773 26991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:21.966050 26991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:21.969609 26991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9319c6b02d844ca8b050fa4d41a36d7e: Sending DeleteTablet for 3 replicas of tablet 5d44cd1cdb1e4958a3cafc6bbb494412
I20260928 15:07:21.970739 27112 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d44cd1cdb1e4958a3cafc6bbb494412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:21 UTC) from {username='slave'} at 127.0.0.1:46454
I20260928 15:07:21.971135 27373 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d44cd1cdb1e4958a3cafc6bbb494412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:21 UTC) from {username='slave'} at 127.0.0.1:54430
I20260928 15:07:21.971179 27243 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d44cd1cdb1e4958a3cafc6bbb494412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:21 UTC) from {username='slave'} at 127.0.0.1:39422
I20260928 15:07:21.971735 27478 tablet_replica.cc:330] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: stopping tablet replica
I20260928 15:07:21.972002 27479 tablet_replica.cc:330] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: stopping tablet replica
I20260928 15:07:21.972306 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27050
I20260928 15:07:21.972389 27478 raft_consensus.cc:2229] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:21.972446 27479 raft_consensus.cc:2229] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:21.973043 27478 raft_consensus.cc:2258] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:21.973037 27479 raft_consensus.cc:2258] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:21.975334 27478 ts_tablet_manager.cc:1905] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:21.983181 27479 ts_tablet_manager.cc:1905] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:21.984069 27478 ts_tablet_manager.cc:1918] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:21.984380 27478 log.cc:1196] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/5d44cd1cdb1e4958a3cafc6bbb494412
I20260928 15:07:21.985041 27478 ts_tablet_manager.cc:1939] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 4bfc5def331241088ebc6d5610e94b15: Deleting consensus metadata
I20260928 15:07:21.986979 26977 catalog_manager.cc:4841] TS 4bfc5def331241088ebc6d5610e94b15 (127.9.12.193:37793): tablet 5d44cd1cdb1e4958a3cafc6bbb494412 (table test-table [id=33e20b83cc584ae5a10366ac6d194d50]) successfully deleted
I20260928 15:07:21.990177 27480 tablet_replica.cc:330] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: stopping tablet replica
I20260928 15:07:21.990703 27480 raft_consensus.cc:2229] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:21.991315 27480 raft_consensus.cc:2258] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:21.993503 27480 ts_tablet_manager.cc:1905] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:21.994086 27479 ts_tablet_manager.cc:1918] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:21.994306 27479 log.cc:1196] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/5d44cd1cdb1e4958a3cafc6bbb494412
I20260928 15:07:21.994920 27479 ts_tablet_manager.cc:1939] T 5d44cd1cdb1e4958a3cafc6bbb494412 P f46cc96154d34338a47d81deefed9371: Deleting consensus metadata
I20260928 15:07:21.996946 26978 catalog_manager.cc:4841] TS f46cc96154d34338a47d81deefed9371 (127.9.12.195:34319): tablet 5d44cd1cdb1e4958a3cafc6bbb494412 (table test-table [id=33e20b83cc584ae5a10366ac6d194d50]) successfully deleted
I20260928 15:07:21.997790 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:21.974364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46468 (local address 127.9.12.193:37793)
0928 15:07:21.974531 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:21.974537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:21.989603 (+ 15066us) server_negotiation.cc:407] Connection header received
0928 15:07:21.990074 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:21.990080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:21.990186 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:21.990274 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:21.990934 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.992005 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.993879 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:21.994472 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:21.994565 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:21.996094 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:21.996105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:21.996123 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:21.996871 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:21.996944 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:21.996972 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:21.997060 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:21.997317 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 15:07:21.997432 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:07:21.999020 27172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:21.999931 27480 ts_tablet_manager.cc:1918] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:22.000098 27480 log.cc:1196] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/5d44cd1cdb1e4958a3cafc6bbb494412
I20260928 15:07:22.000638 27480 ts_tablet_manager.cc:1939] T 5d44cd1cdb1e4958a3cafc6bbb494412 P 0c76176663fc4a32b5c8fd0c38742fe7: Deleting consensus metadata
I20260928 15:07:22.002142 26978 catalog_manager.cc:4841] TS 0c76176663fc4a32b5c8fd0c38742fe7 (127.9.12.194:45411): tablet 5d44cd1cdb1e4958a3cafc6bbb494412 (table test-table [id=33e20b83cc584ae5a10366ac6d194d50]) successfully deleted
I20260928 15:07:22.116382 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27050
I20260928 15:07:22.140738 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27181
I20260928 15:07:22.149911 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:22.141947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.9.12.194:45411)
0928 15:07:22.142163 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:22.142168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:22.142207 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:07:22.142276 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:22.142285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:22.142412 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:22.142524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:22.143235 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.144476 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.146096 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.146751 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.146910 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:22.148161 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:22.148172 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:22.148193 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:22.149025 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:22.149094 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:22.149122 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:22.149247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:22.149420 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 15:07:22.149541 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:07:22.150660 27303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:22.241988 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27181
I20260928 15:07:22.260383 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27311
I20260928 15:07:22.268663 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:22.261267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.9.12.195:34319)
0928 15:07:22.261452 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:22.261457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:22.261514 (+ 57us) server_negotiation.cc:407] Connection header received
0928 15:07:22.261711 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:22.261715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:22.261795 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:22.261898 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:22.262397 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.263214 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.265339 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.265795 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.265897 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:22.267144 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:22.267153 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:22.267167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:22.267772 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:22.267832 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:22.267853 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:22.267936 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:22.268122 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 15:07:22.268207 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:07:22.269459 27433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:22.391055 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27311
I20260928 15:07:22.410780 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 26960
I20260928 15:07:22.411923 27021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:22.525701 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 26960
2026-09-28T15:07:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2848 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:22 dist-test-slave-dmhs krb5kdc[27488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:22 dist-test-slave-dmhs krb5kdc[27488](info): set up 2 sockets
Sep 28 15:07:22 dist-test-slave-dmhs krb5kdc[27488](info): commencing operation
krb5kdc: starting...
W20260928 15:07:23.628826 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:23 dist-test-slave-dmhs krb5kdc[27488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:07:23.885931 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:36005
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:36005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:24.214160 27501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:24.214529 27501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:24.214708 27501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:24.221853 27501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:24.222004 27501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:24.222113 27501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:24.222216 27501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:24.222302 27501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:24.232050 27501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:36005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:36005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:24.233409 27501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:24.234746 27501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:24.247924 27507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:24.247848 27509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:24.247812 27506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:24.249143 27501 server_base.cc:1034] running on GCE node
I20260928 15:07:24.249995 27501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:24.251515 27501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:24.252743 27501 hybrid_clock.cc:648] HybridClock initialized: now 1790608044252699 us; error 55 us; skew 500 ppm
I20260928 15:07:24.253105 27501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:24 dist-test-slave-dmhs krb5kdc[27488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608044, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:24.256158 27501 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:07:24.257709 27501 webserver.cc:460] Webserver started at http://127.9.12.254:33775/ using document root <none> and password file <none>
I20260928 15:07:24.258394 27501 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:24.258549 27501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:24.258863 27501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:24.261973 27501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "47bf231034e54bdb82b48c3ef4faaa9a"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-dmhs"
server_key: "8ad4214c90a5923edd46ac9ee421349f"
server_key_iv: "c5c5af92d8500f58f6fcd17571140879"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.262738 27501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "47bf231034e54bdb82b48c3ef4faaa9a"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-dmhs"
server_key: "8ad4214c90a5923edd46ac9ee421349f"
server_key_iv: "c5c5af92d8500f58f6fcd17571140879"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.268024 27501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:07:24.271565 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:24.272907 27501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:24.273094 27501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "47bf231034e54bdb82b48c3ef4faaa9a"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-dmhs"
server_key: "8ad4214c90a5923edd46ac9ee421349f"
server_key_iv: "c5c5af92d8500f58f6fcd17571140879"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.273303 27501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:24.303610 27501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:24.307812 27501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:24.308131 27501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:24.327204 27501 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:36005
I20260928 15:07:24.327236 27567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:36005 every 8 connection(s)
I20260928 15:07:24.328722 27501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:24.330492 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27501
I20260928 15:07:24.330687 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:24.331166 9267 external_mini_cluster.cc:1428] Setting key a0fe0b66ba8fb814f76c86b4ce0b1eb5
I20260928 15:07:24.332967 27568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:07:24 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:24.346942 27568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a: Bootstrap starting.
I20260928 15:07:24.350167 27568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:24.351249 27568 log.cc:824] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:24.354218 27568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a: No bootstrap required, opened a new log
I20260928 15:07:24.354660 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:24.333019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.9.12.254:36005)
0928 15:07:24.333846 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:24.333857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:24.333897 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:07:24.335530 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:24.335552 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:24.335930 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:24.336255 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:24.338151 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:24.339485 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:24.341438 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:24.342150 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:24.342311 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:24.347233 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:24.347241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:24.347257 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:24.347295 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:24.350517 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:24.351239 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:24.351246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:24.351252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:24.351454 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:24.351847 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:24.351852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:24.351856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:24.352463 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:24.352776 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:24.353226 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 15:07:24.353649 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":362,"threads_started":1}
I20260928 15:07:24.362078 27568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36005 } }
I20260928 15:07:24.362511 27568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:24.362643 27568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bf231034e54bdb82b48c3ef4faaa9a, State: Initialized, Role: FOLLOWER
I20260928 15:07:24.363226 27568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36005 } }
I20260928 15:07:24.363472 27568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:24.363621 27568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:24.363792 27568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:24.365975 27568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36005 } }
I20260928 15:07:24.366451 27568 leader_election.cc:302] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47bf231034e54bdb82b48c3ef4faaa9a; no voters:
I20260928 15:07:24.367110 27568 leader_election.cc:288] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:24.367237 27573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:24.367911 27573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [term 1 LEADER]: Becoming Leader. State: Replica: 47bf231034e54bdb82b48c3ef4faaa9a, State: Running, Role: LEADER
I20260928 15:07:24.368535 27573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36005 } }
I20260928 15:07:24.369254 27568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:24.371093 27574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36005 } } }
I20260928 15:07:24.371516 27574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:24.372507 27579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:24.372459 27575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47bf231034e54bdb82b48c3ef4faaa9a. Latest consensus state: current_term: 1 leader_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bf231034e54bdb82b48c3ef4faaa9a" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36005 } } }
I20260928 15:07:24.374131 27575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:24.377970 27579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:24.387449 27579 catalog_manager.cc:1347] Generated new cluster ID: dae5782707404fa681b7900de187c740
I20260928 15:07:24.387562 27579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:24.414570 27579 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:24.415926 27579 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:24.430478 27579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a: Generated new TSK 0
I20260928 15:07:24.431319 27579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:07:24.500717 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36005
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:24.854617 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:24.854907 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:24.855002 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:24.862363 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:24.862479 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:24.862594 27594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:07:24.872757 27594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:24.874055 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:24.875317 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:24.889053 27599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:24.889097 27602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:24.889070 27600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:24.889937 27594 server_base.cc:1034] running on GCE node
I20260928 15:07:24.890568 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:24.891538 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:24.892771 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790608044892728 us; error 52 us; skew 500 ppm
I20260928 15:07:24.893129 27594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:24 dist-test-slave-dmhs krb5kdc[27488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608044, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:24.896209 27594 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:07:24.897734 27594 webserver.cc:460] Webserver started at http://127.9.12.193:33577/ using document root <none> and password file <none>
I20260928 15:07:24.898520 27594 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:24.898671 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:24.898994 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:24.901961 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "924457e33870432492391fcc4290d260"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-dmhs"
server_key: "3947635e915d18464373b85a6a5d07cb"
server_key_iv: "65052e9d056889d7775042c1373a4491"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.902757 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "924457e33870432492391fcc4290d260"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-dmhs"
server_key: "3947635e915d18464373b85a6a5d07cb"
server_key_iv: "65052e9d056889d7775042c1373a4491"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.907882 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 15:07:24.911404 27609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:24.912860 27594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:24.913074 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "924457e33870432492391fcc4290d260"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-dmhs"
server_key: "3947635e915d18464373b85a6a5d07cb"
server_key_iv: "65052e9d056889d7775042c1373a4491"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.913302 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:24.929123 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:24.933136 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:24.933446 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:24.934414 27594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:24.935768 27594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:24.935874 27594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:24.936036 27594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:24.936110 27594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:24.970345 27594 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:44391
I20260928 15:07:24.970409 27721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:44391 every 8 connection(s)
I20260928 15:07:24.971692 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:24.980770 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27594
I20260928 15:07:24.980971 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:24.981451 9267 external_mini_cluster.cc:1428] Setting key 136d4974bb77326c6959927040772de1
Sep 28 15:07:24 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608044, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:24.992213 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:24.974403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51445 (local address 127.9.12.254:36005)
0928 15:07:24.974870 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:24.974874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:24.975884 (+ 1010us) server_negotiation.cc:407] Connection header received
0928 15:07:24.977995 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:24.978001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:24.978088 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:24.978206 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:24.979814 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:24.980624 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:24.982383 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:24.983013 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:24.983160 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:24.986619 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:24.986623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:24.986627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:24.986657 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:24.989217 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:24.989936 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:24.989944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:24.989949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:24.990070 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:24.990509 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:24.990515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:24.990518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:24.990794 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:24.990947 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:24.991793 (+ 846us) server_negotiation.cc:299] Negotiation successful
0928 15:07:24.991968 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":224,"threads_started":1}
I20260928 15:07:24.993768 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:24.974946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36005 (local address 127.9.12.193:51445)
0928 15:07:24.975636 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:24.975674 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:24.977697 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:24.978454 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:24.978462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:24.978998 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:24.979638 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:24.979657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:24.980781 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:24.980785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:24.982202 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:24.982218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:24.983272 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:24.983280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:24.983516 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:24.984210 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:24.984236 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:24.986402 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:24.989356 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:24.989365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:24.989382 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:24.989805 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:24.990188 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:24.990192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:24.990196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:24.990395 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:24.990926 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:24.990934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:24.991371 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 15:07:24.991701 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 15:07:24.992307 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
I20260928 15:07:24.995399 27722 heartbeater.cc:347] Connected to a master server at 127.9.12.254:36005
I20260928 15:07:24.995711 27722 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:24.996503 27722 heartbeater.cc:511] Master 127.9.12.254:36005 requested a full tablet report, sending...
I20260928 15:07:24.998656 27533 ts_manager.cc:194] Registered new tserver with Master: 924457e33870432492391fcc4290d260 (127.9.12.193:44391)
I20260928 15:07:25.000104 27533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:51445
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:07:25.038971 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36005
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:25.370680 27728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:25.371011 27728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:25.371186 27728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:25.379310 27728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:25.379479 27728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:25.379655 27728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:07:25.389559 27728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:25.390904 27728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:25.392243 27728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:25.405652 27736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:25.405655 27733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:25.405722 27734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:25.406342 27728 server_base.cc:1034] running on GCE node
I20260928 15:07:25.406984 27728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:25.407866 27728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:25.409065 27728 hybrid_clock.cc:648] HybridClock initialized: now 1790608045409026 us; error 34 us; skew 500 ppm
I20260928 15:07:25.409379 27728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:25 dist-test-slave-dmhs krb5kdc[27488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:25.412523 27728 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:07:25.413887 27728 webserver.cc:460] Webserver started at http://127.9.12.194:32915/ using document root <none> and password file <none>
I20260928 15:07:25.414525 27728 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:25.414625 27728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:25.414882 27728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:25.417917 27728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "e9fd6c3b8759420083c9ae00c04af44e"
format_stamp: "Formatted at 2026-09-28 15:07:25 on dist-test-slave-dmhs"
server_key: "897a3ca7bf2822da69dabe8e7e2e1475"
server_key_iv: "9c2134a9f93fb824eb12188832eeeecb"
server_key_version: "encryptionkey@0"
I20260928 15:07:25.418639 27728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "e9fd6c3b8759420083c9ae00c04af44e"
format_stamp: "Formatted at 2026-09-28 15:07:25 on dist-test-slave-dmhs"
server_key: "897a3ca7bf2822da69dabe8e7e2e1475"
server_key_iv: "9c2134a9f93fb824eb12188832eeeecb"
server_key_version: "encryptionkey@0"
I20260928 15:07:25.423710 27728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 15:07:25.427086 27743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:25.428452 27728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:25.428617 27728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "e9fd6c3b8759420083c9ae00c04af44e"
format_stamp: "Formatted at 2026-09-28 15:07:25 on dist-test-slave-dmhs"
server_key: "897a3ca7bf2822da69dabe8e7e2e1475"
server_key_iv: "9c2134a9f93fb824eb12188832eeeecb"
server_key_version: "encryptionkey@0"
I20260928 15:07:25.428821 27728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:25.454427 27728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:25.458671 27728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:25.458992 27728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:25.460016 27728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:25.461400 27728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:25.461504 27728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:25.461658 27728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:25.461732 27728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:25.495420 27728 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:37093
I20260928 15:07:25.495473 27855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:37093 every 8 connection(s)
I20260928 15:07:25.496833 27728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:25.503623 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27728
I20260928 15:07:25.503808 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:25.504418 9267 external_mini_cluster.cc:1428] Setting key a350168d950208f043f094a454043e5f
Sep 28 15:07:25 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:25.519001 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:25.499439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:58019 (local address 127.9.12.254:36005)
0928 15:07:25.499931 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:25.499936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:25.500851 (+ 915us) server_negotiation.cc:407] Connection header received
0928 15:07:25.502728 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:25.502734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:25.502828 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:25.502953 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:25.505122 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:25.505921 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:25.508217 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:25.508701 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:25.508804 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:25.513048 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:25.513053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:25.513056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:25.513088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:25.515661 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:25.516423 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:25.516433 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:25.516436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:25.516601 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:25.517086 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:25.517091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:25.517094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:25.517375 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:25.517533 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:25.518360 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 15:07:25.518590 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":243,"threads_started":1}
I20260928 15:07:25.519979 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:25.499913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36005 (local address 127.9.12.194:58019)
0928 15:07:25.500684 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:25.500727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:25.502450 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:25.503214 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:25.503226 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:25.504026 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:25.504871 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:25.504911 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:25.506071 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:25.506079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:25.508066 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:25.508090 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:25.508931 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:25.508940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:25.509259 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:25.510004 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:25.510037 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:25.512815 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:25.515803 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:25.515812 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:25.515830 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:25.516310 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:25.516716 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:25.516722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:25.516725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:25.516962 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:25.517509 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:25.517517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:25.517948 (+ 431us) client_negotiation.cc:769] Sending connection context
0928 15:07:25.518257 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 15:07:25.518807 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":329,"threads_started":1}
I20260928 15:07:25.521571 27856 heartbeater.cc:347] Connected to a master server at 127.9.12.254:36005
I20260928 15:07:25.521895 27856 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:25.522763 27856 heartbeater.cc:511] Master 127.9.12.254:36005 requested a full tablet report, sending...
I20260928 15:07:25.524742 27533 ts_manager.cc:194] Registered new tserver with Master: e9fd6c3b8759420083c9ae00c04af44e (127.9.12.194:37093)
I20260928 15:07:25.526012 27533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:58019
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:07:25.564141 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36005
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:25.895272 27862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:25.895552 27862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:25.895648 27862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:25.903080 27862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:25.903199 27862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:25.903316 27862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:07:25.913286 27862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:25.914563 27862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:25.915860 27862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:25.929572 27870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:25.929688 27867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:25.929524 27868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:25.930526 27862 server_base.cc:1034] running on GCE node
I20260928 15:07:25.931130 27862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:25.932142 27862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:25.933341 27862 hybrid_clock.cc:648] HybridClock initialized: now 1790608045933308 us; error 52 us; skew 500 ppm
I20260928 15:07:25.933677 27862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:25 dist-test-slave-dmhs krb5kdc[27488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:25.936914 27862 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:07:25.938436 27862 webserver.cc:460] Webserver started at http://127.9.12.195:46039/ using document root <none> and password file <none>
I20260928 15:07:25.939129 27862 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:25.939275 27862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:25.939591 27862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:25.942569 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "51d5789efc8a4960b0cdb8aaebfe7375"
format_stamp: "Formatted at 2026-09-28 15:07:25 on dist-test-slave-dmhs"
server_key: "cd472431e064b61b0dbd6e66ded8bcbc"
server_key_iv: "2bb31211f446ad520ddc0f5ca6843dec"
server_key_version: "encryptionkey@0"
I20260928 15:07:25.943400 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "51d5789efc8a4960b0cdb8aaebfe7375"
format_stamp: "Formatted at 2026-09-28 15:07:25 on dist-test-slave-dmhs"
server_key: "cd472431e064b61b0dbd6e66ded8bcbc"
server_key_iv: "2bb31211f446ad520ddc0f5ca6843dec"
server_key_version: "encryptionkey@0"
I20260928 15:07:25.948972 27862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:07:25.952621 27877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:25.954054 27862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:07:25.954257 27862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "51d5789efc8a4960b0cdb8aaebfe7375"
format_stamp: "Formatted at 2026-09-28 15:07:25 on dist-test-slave-dmhs"
server_key: "cd472431e064b61b0dbd6e66ded8bcbc"
server_key_iv: "2bb31211f446ad520ddc0f5ca6843dec"
server_key_version: "encryptionkey@0"
I20260928 15:07:25.954487 27862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:25.997699 27862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:26.002485 27862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:26.002840 27862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:26.003033 27722 heartbeater.cc:503] Master 127.9.12.254:36005 was elected leader, sending a full tablet report...
I20260928 15:07:26.004148 27862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:26.005959 27862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:26.006035 27862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:26.006126 27862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:26.006172 27862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:26.040194 27862 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:35781
I20260928 15:07:26.040347 27989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:35781 every 8 connection(s)
I20260928 15:07:26.041587 27862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:26.050626 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 27862
I20260928 15:07:26.050876 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:26.051347 9267 external_mini_cluster.cc:1428] Setting key e76d0e1bca4e9c312797444cf4f29696
Sep 28 15:07:26 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:26.062047 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.044243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:45375 (local address 127.9.12.254:36005)
0928 15:07:26.044810 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.044839 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.045647 (+ 808us) server_negotiation.cc:407] Connection header received
0928 15:07:26.047540 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.047547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.047654 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.047777 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:26.049424 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.050206 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.051915 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.052600 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.052737 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.056292 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:26.056299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:26.056304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:26.056349 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:26.058862 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.059600 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.059607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.059610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.059744 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.060225 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.060231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.060234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.060574 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:26.060747 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.061420 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.061645 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":282,"threads_started":1}
I20260928 15:07:26.063012 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.044780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36005 (local address 127.9.12.195:45375)
0928 15:07:26.045479 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:26.045527 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:26.047251 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:26.048035 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:26.048044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:26.048660 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:26.049240 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.049259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.050361 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.050368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.051771 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.051783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.052836 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.052844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.053076 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.053730 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:26.053753 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:26.056048 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:26.059027 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:26.059036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:26.059060 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:26.059477 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:26.059863 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:26.059869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:26.059872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:26.060115 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:26.060713 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:26.060722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:26.061066 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 15:07:26.061345 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 15:07:26.061869 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":285,"threads_started":1}
I20260928 15:07:26.064532 27990 heartbeater.cc:347] Connected to a master server at 127.9.12.254:36005
I20260928 15:07:26.064868 27990 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:26.065795 27990 heartbeater.cc:511] Master 127.9.12.254:36005 requested a full tablet report, sending...
I20260928 15:07:26.067343 27533 ts_manager.cc:194] Registered new tserver with Master: 51d5789efc8a4960b0cdb8aaebfe7375 (127.9.12.195:35781)
I20260928 15:07:26.068156 27533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:45375
I20260928 15:07:26.079401 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:26.097201 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.083787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.9.12.254:36005)
0928 15:07:26.083985 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.083993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.084397 (+ 404us) server_negotiation.cc:407] Connection header received
0928 15:07:26.084606 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.084612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.084696 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.084801 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:26.086233 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.086996 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.088436 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.088852 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.089000 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.091781 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:26.091787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:26.091790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:26.091826 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:26.094275 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.095074 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.095080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.095085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.095214 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.095680 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.095684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.095686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.095997 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:26.096303 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.096709 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.096909 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:07:26.101292 27533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:26.108336 27533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:26.146927 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.121825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.9.12.193:44391)
0928 15:07:26.131971 (+ 10146us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.131985 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.135425 (+ 3440us) server_negotiation.cc:407] Connection header received
0928 15:07:26.135575 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.135588 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.135953 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.136178 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:26.138843 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.141528 (+ 2685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.144448 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.145673 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.145824 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.146100 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.146376 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.146555 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9911,"thread_start_us":9718,"threads_started":1}
I20260928 15:07:26.146993 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.121237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:44391 (local address 127.0.0.1:48548)
0928 15:07:26.134793 (+ 13556us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:26.134857 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:26.135116 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:26.137962 (+ 2846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:26.137973 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:26.138020 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:26.138582 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.138597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.141670 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.141675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.144166 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.144188 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.145886 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.145893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.146105 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.146168 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 15:07:26.146320 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:07:26.146532 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13192,"mutex_wait_us":5,"thread_start_us":18437,"threads_started":1}
I20260928 15:07:26.147965 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.121861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.9.12.195:35781)
0928 15:07:26.132989 (+ 11128us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.133000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.139413 (+ 6413us) server_negotiation.cc:407] Connection header received
0928 15:07:26.140029 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.140037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.140326 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.140519 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:26.141445 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.142702 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.144994 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.146466 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.146561 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.146781 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.147414 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.147595 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10439,"thread_start_us":10291,"threads_started":1}
I20260928 15:07:26.148793 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.121291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:35781 (local address 127.0.0.1:35462)
0928 15:07:26.139206 (+ 17915us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:26.139245 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:26.139826 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:26.140791 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:26.140796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:26.140822 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:26.141210 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.141221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.142846 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.142854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.144805 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.144821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.147001 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.147008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.147206 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.147241 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:07:26.148238 (+ 997us) client_negotiation.cc:240] Negotiation successful
0928 15:07:26.148414 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17714,"thread_start_us":18174,"threads_started":1}
I20260928 15:07:26.149246 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.121757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:37093 (local address 127.0.0.1:38380)
0928 15:07:26.139465 (+ 17708us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:26.139489 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:26.139606 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:26.141275 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:26.141278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:26.141299 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:26.141805 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.141818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.143946 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.143951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.146188 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.146203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.148408 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.148416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.148601 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.148633 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:07:26.148732 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:07:26.148872 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17580,"thread_start_us":14737,"threads_started":1}
I20260928 15:07:26.150066 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.122007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.9.12.194:37093)
0928 15:07:26.140437 (+ 18430us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.140446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.140475 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:07:26.140570 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.140576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.140800 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.140988 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:26.142006 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.143599 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.146395 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.147958 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.148124 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.148337 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.149524 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.149704 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18237,"thread_start_us":18113,"threads_started":1}
I20260928 15:07:26.151593 27657 tablet_service.cc:1467] Processing CreateTablet for tablet 4681188551134b9f8b9b2024e2445ab1 (DEFAULT_TABLE table=test-table [id=040281074c69476bbeefec7c6c9cb814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:26.152465 27925 tablet_service.cc:1467] Processing CreateTablet for tablet 4681188551134b9f8b9b2024e2445ab1 (DEFAULT_TABLE table=test-table [id=040281074c69476bbeefec7c6c9cb814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:26.153245 27657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4681188551134b9f8b9b2024e2445ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:26.153679 27925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4681188551134b9f8b9b2024e2445ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:26.153864 27791 tablet_service.cc:1467] Processing CreateTablet for tablet 4681188551134b9f8b9b2024e2445ab1 (DEFAULT_TABLE table=test-table [id=040281074c69476bbeefec7c6c9cb814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:26.155146 27791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4681188551134b9f8b9b2024e2445ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:26.165016 28009 tablet_bootstrap.cc:492] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Bootstrap starting.
I20260928 15:07:26.168978 28009 tablet_bootstrap.cc:654] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:26.168965 28010 tablet_bootstrap.cc:492] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Bootstrap starting.
I20260928 15:07:26.170486 28009 log.cc:824] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:26.170807 28011 tablet_bootstrap.cc:492] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Bootstrap starting.
I20260928 15:07:26.172559 28010 tablet_bootstrap.cc:654] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:26.173779 28010 log.cc:824] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:26.174036 28011 tablet_bootstrap.cc:654] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:26.174541 28009 tablet_bootstrap.cc:492] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: No bootstrap required, opened a new log
I20260928 15:07:26.174814 28009 ts_tablet_manager.cc:1397] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:07:26.175287 28011 log.cc:824] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:26.177814 28010 tablet_bootstrap.cc:492] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: No bootstrap required, opened a new log
I20260928 15:07:26.178120 28010 ts_tablet_manager.cc:1397] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 15:07:26.178344 28011 tablet_bootstrap.cc:492] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: No bootstrap required, opened a new log
I20260928 15:07:26.178598 28011 ts_tablet_manager.cc:1397] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 15:07:26.185907 28011 raft_consensus.cc:348] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.186324 28011 raft_consensus.cc:374] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:26.186453 28011 raft_consensus.cc:729] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd6c3b8759420083c9ae00c04af44e, State: Initialized, Role: FOLLOWER
I20260928 15:07:26.186380 28009 raft_consensus.cc:348] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.186839 28009 raft_consensus.cc:374] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:26.186951 28009 raft_consensus.cc:729] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d5789efc8a4960b0cdb8aaebfe7375, State: Initialized, Role: FOLLOWER
I20260928 15:07:26.187304 28011 consensus_queue.cc:260] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.187592 28009 consensus_queue.cc:260] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.190366 28010 raft_consensus.cc:348] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.190912 28010 raft_consensus.cc:374] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:26.191033 28010 raft_consensus.cc:729] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924457e33870432492391fcc4290d260, State: Initialized, Role: FOLLOWER
I20260928 15:07:26.191747 28010 consensus_queue.cc:260] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.192250 27990 heartbeater.cc:503] Master 127.9.12.254:36005 was elected leader, sending a full tablet report...
I20260928 15:07:26.193378 28009 ts_tablet_manager.cc:1428] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260928 15:07:26.194424 27856 heartbeater.cc:503] Master 127.9.12.254:36005 was elected leader, sending a full tablet report...
I20260928 15:07:26.194979 28011 ts_tablet_manager.cc:1428] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260928 15:07:26.197315 28010 ts_tablet_manager.cc:1428] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
W20260928 15:07:26.224201 27723 tablet.cc:2369] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:26.249009 27857 tablet.cc:2369] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:26.293284 27991 tablet.cc:2369] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:26.573791 28016 raft_consensus.cc:482] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:26.574133 28016 raft_consensus.cc:504] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.575704 28016 leader_election.cc:288] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 924457e33870432492391fcc4290d260 (127.9.12.193:44391), e9fd6c3b8759420083c9ae00c04af44e (127.9.12.194:37093)
I20260928 15:07:26.584770 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.576221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:41581 (local address 127.9.12.193:44391)
0928 15:07:26.576443 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.576450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.577056 (+ 606us) server_negotiation.cc:407] Connection header received
0928 15:07:26.577344 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.577349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.577428 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.577535 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:26.578456 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.579513 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.582380 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.583638 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.583805 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.583883 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.584369 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.584503 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:07:26.584764 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.576177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:44403 (local address 127.9.12.194:37093)
0928 15:07:26.576411 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.576417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.577101 (+ 684us) server_negotiation.cc:407] Connection header received
0928 15:07:26.577341 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.577346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.577439 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.577547 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:26.578293 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.579457 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.582399 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.583662 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.583805 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.583883 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.584368 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.584511 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:07:26.585045 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.576012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:44391 (local address 127.9.12.195:41581)
0928 15:07:26.576898 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:26.576927 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:26.577185 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:26.577570 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:26.577575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:26.577605 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:26.578282 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.578295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.579628 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.579631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.582146 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.582162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.583837 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.583849 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.584112 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.584210 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 15:07:26.584350 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:07:26.584533 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":188,"spinlock_wait_cycles":23296,"thread_start_us":596,"threads_started":1}
I20260928 15:07:26.585071 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.576061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:37093 (local address 127.9.12.195:44403)
0928 15:07:26.576987 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:26.577008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:26.577150 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:26.577570 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:26.577575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:26.577634 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:26.578043 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.578108 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.579621 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.579626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.582146 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:26.582162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.583861 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.583866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:26.584116 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.584215 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 15:07:26.584342 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:07:26.584534 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":177,"spinlock_wait_cycles":28544,"thread_start_us":554,"threads_started":1}
I20260928 15:07:26.586179 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4681188551134b9f8b9b2024e2445ab1" candidate_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fd6c3b8759420083c9ae00c04af44e" is_pre_election: true
I20260928 15:07:26.586274 27677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4681188551134b9f8b9b2024e2445ab1" candidate_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924457e33870432492391fcc4290d260" is_pre_election: true
I20260928 15:07:26.586706 27811 raft_consensus.cc:2454] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51d5789efc8a4960b0cdb8aaebfe7375 in term 0.
I20260928 15:07:26.586757 27677 raft_consensus.cc:2454] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51d5789efc8a4960b0cdb8aaebfe7375 in term 0.
I20260928 15:07:26.587451 27879 leader_election.cc:302] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51d5789efc8a4960b0cdb8aaebfe7375, 924457e33870432492391fcc4290d260, e9fd6c3b8759420083c9ae00c04af44e; no voters:
I20260928 15:07:26.587888 28016 raft_consensus.cc:2781] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:26.588028 28016 raft_consensus.cc:482] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:26.588135 28016 raft_consensus.cc:3037] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:26.590376 28016 raft_consensus.cc:504] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.591069 28016 leader_election.cc:288] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [CANDIDATE]: Term 1 election: Requested vote from peers 924457e33870432492391fcc4290d260 (127.9.12.193:44391), e9fd6c3b8759420083c9ae00c04af44e (127.9.12.194:37093)
I20260928 15:07:26.591441 27677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4681188551134b9f8b9b2024e2445ab1" candidate_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924457e33870432492391fcc4290d260"
I20260928 15:07:26.591530 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4681188551134b9f8b9b2024e2445ab1" candidate_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fd6c3b8759420083c9ae00c04af44e"
I20260928 15:07:26.591655 27677 raft_consensus.cc:3037] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:26.591761 27811 raft_consensus.cc:3037] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:26.594444 27677 raft_consensus.cc:2454] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d5789efc8a4960b0cdb8aaebfe7375 in term 1.
I20260928 15:07:26.594463 27811 raft_consensus.cc:2454] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d5789efc8a4960b0cdb8aaebfe7375 in term 1.
I20260928 15:07:26.594862 27880 leader_election.cc:302] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51d5789efc8a4960b0cdb8aaebfe7375, 924457e33870432492391fcc4290d260; no voters:
I20260928 15:07:26.595156 28016 raft_consensus.cc:2781] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:26.595618 28016 raft_consensus.cc:686] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 1 LEADER]: Becoming Leader. State: Replica: 51d5789efc8a4960b0cdb8aaebfe7375, State: Running, Role: LEADER
I20260928 15:07:26.596081 28016 consensus_queue.cc:237] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } }
I20260928 15:07:26.602140 27532 catalog_manager.cc:5495] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51d5789efc8a4960b0cdb8aaebfe7375 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51d5789efc8a4960b0cdb8aaebfe7375" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 35781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 } health_report { overall_health: UNKNOWN } } }
I20260928 15:07:26.716061 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.709958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.9.12.195:35781)
0928 15:07:26.710464 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.710471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.710650 (+ 179us) server_negotiation.cc:407] Connection header received
0928 15:07:26.710893 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.710897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.710971 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.711135 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:26.711593 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.712256 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.713826 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.714237 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.714362 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.714973 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:26.715460 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:26.715674 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.715759 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.715806 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260928 15:07:26.724115 27677 raft_consensus.cc:1261] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 1 FOLLOWER]: Refusing update from remote peer 51d5789efc8a4960b0cdb8aaebfe7375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:26.724115 27811 raft_consensus.cc:1261] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 1 FOLLOWER]: Refusing update from remote peer 51d5789efc8a4960b0cdb8aaebfe7375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:26.725147 28020 consensus_queue.cc:1035] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fd6c3b8759420083c9ae00c04af44e" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:26.725586 28016 consensus_queue.cc:1035] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "924457e33870432492391fcc4290d260" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:26.736680 28025 mvcc.cc:205] Tried to move back new op lower bound from 7334330559371767808 to 7334330558861529088. Current Snapshot: MvccSnapshot[applied={T|T < 7334330559371767808}]
I20260928 15:07:26.751477 27532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:26.751770 27532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:26.755580 27532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47bf231034e54bdb82b48c3ef4faaa9a: Sending DeleteTablet for 3 replicas of tablet 4681188551134b9f8b9b2024e2445ab1
I20260928 15:07:26.756826 27925 tablet_service.cc:1514] Processing DeleteTablet for tablet 4681188551134b9f8b9b2024e2445ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:26 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:35462
I20260928 15:07:26.756820 27657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4681188551134b9f8b9b2024e2445ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:26 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:48548
I20260928 15:07:26.756920 27791 tablet_service.cc:1514] Processing DeleteTablet for tablet 4681188551134b9f8b9b2024e2445ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:26 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:38380
I20260928 15:07:26.761385 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27594
I20260928 15:07:26.769613 28039 tablet_replica.cc:330] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: stopping tablet replica
I20260928 15:07:26.770287 28039 raft_consensus.cc:2229] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:26.770767 28038 tablet_replica.cc:330] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: stopping tablet replica
I20260928 15:07:26.770985 28039 raft_consensus.cc:2258] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:26.771256 28038 raft_consensus.cc:2229] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:26.771799 28038 raft_consensus.cc:2258] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:26.773166 28039 ts_tablet_manager.cc:1905] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:26.773825 28038 ts_tablet_manager.cc:1905] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:26.780323 28037 tablet_replica.cc:330] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: stopping tablet replica
I20260928 15:07:26.780822 28037 raft_consensus.cc:2229] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:26.780941 28039 ts_tablet_manager.cc:1918] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 15:07:26 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:07:26.781189 28039 log.cc:1196] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/4681188551134b9f8b9b2024e2445ab1
I20260928 15:07:26.781383 28038 ts_tablet_manager.cc:1918] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:26.781582 28037 raft_consensus.cc:2258] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:26.781651 28038 log.cc:1196] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/4681188551134b9f8b9b2024e2445ab1
I20260928 15:07:26.781852 28039 ts_tablet_manager.cc:1939] T 4681188551134b9f8b9b2024e2445ab1 P e9fd6c3b8759420083c9ae00c04af44e: Deleting consensus metadata
I20260928 15:07:26.782297 28038 ts_tablet_manager.cc:1939] T 4681188551134b9f8b9b2024e2445ab1 P 924457e33870432492391fcc4290d260: Deleting consensus metadata
I20260928 15:07:26.783619 28037 ts_tablet_manager.cc:1905] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:26.784343 27519 catalog_manager.cc:4841] TS 924457e33870432492391fcc4290d260 (127.9.12.193:44391): tablet 4681188551134b9f8b9b2024e2445ab1 (table test-table [id=040281074c69476bbeefec7c6c9cb814]) successfully deleted
I20260928 15:07:26.784371 27518 catalog_manager.cc:4841] TS e9fd6c3b8759420083c9ae00c04af44e (127.9.12.194:37093): tablet 4681188551134b9f8b9b2024e2445ab1 (table test-table [id=040281074c69476bbeefec7c6c9cb814]) successfully deleted
I20260928 15:07:26.790827 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.762850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.9.12.193:44391)
0928 15:07:26.763047 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.763054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.769714 (+ 6660us) server_negotiation.cc:407] Connection header received
0928 15:07:26.769984 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.769993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.770099 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.770198 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:26.772286 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.773464 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.775476 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.776215 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.776400 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.782364 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:26.782380 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:26.782399 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:26.782463 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:26.787110 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.788106 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.788119 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.788125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.788324 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.788815 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.788820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.788824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.789265 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:26.789566 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.790188 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.790453 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:07:26.791478 28037 ts_tablet_manager.cc:1918] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:26.791649 28037 log.cc:1196] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/4681188551134b9f8b9b2024e2445ab1
I20260928 15:07:26.792256 28037 ts_tablet_manager.cc:1939] T 4681188551134b9f8b9b2024e2445ab1 P 51d5789efc8a4960b0cdb8aaebfe7375: Deleting consensus metadata
I20260928 15:07:26.792583 27717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:26.794416 27517 catalog_manager.cc:4841] TS 51d5789efc8a4960b0cdb8aaebfe7375 (127.9.12.195:35781): tablet 4681188551134b9f8b9b2024e2445ab1 (table test-table [id=040281074c69476bbeefec7c6c9cb814]) successfully deleted
I20260928 15:07:26.885967 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27594
I20260928 15:07:26.904785 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27728
Sep 28 15:07:26 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:07:26.922366 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.905725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.9.12.194:37093)
0928 15:07:26.905901 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.905905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.905923 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:07:26.906041 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.906044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.906108 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.906190 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:26.907901 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.908747 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.910618 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.911037 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.911139 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.916609 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:26.916627 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:26.916643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:26.916682 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:26.919469 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.920203 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.920210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.920212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.920403 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.920790 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.920796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.920799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.921189 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:26.921501 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.921798 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.922024 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:07:26.923110 27851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:27.013801 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27728
I20260928 15:07:27.031359 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27862
Sep 28 15:07:27 dist-test-slave-dmhs krb5kdc[27488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:07:27.057626 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:27.032388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.9.12.195:35781)
0928 15:07:27.032593 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:27.032600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:27.032629 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:07:27.032722 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:27.032726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:27.032799 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:27.032921 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:27.034341 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.035142 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.036615 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.037094 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.037278 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:27.049624 (+ 12346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:27.049642 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:27.049660 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:27.049734 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:27.054036 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:27.055033 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:27.055049 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:27.055057 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:27.055239 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:27.055671 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:27.055676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:27.055680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:27.056175 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:27.056512 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:27.056911 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 15:07:27.057222 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:07:27.058427 27985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:27.185931 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27862
I20260928 15:07:27.205533 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 27501
I20260928 15:07:27.206710 27563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:27.323321 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 27501
2026-09-28T15:07:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4795 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T15:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:27Z Disabled control of system clock
I20260928 15:07:27.402696 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42971
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:42971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:27.738529 28048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:27.738837 28048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:27.738978 28048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:27.746464 28048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:27.746587 28048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:27.746659 28048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:27.746764 28048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:27.746843 28048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:27.756423 28048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:42971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:42971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:27.757848 28048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:27.759797 28048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:27.772490 28056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:27.772459 28053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:27.772531 28054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:27.773535 28048 server_base.cc:1034] running on GCE node
I20260928 15:07:27.774286 28048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:27.775717 28048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:27.776959 28048 hybrid_clock.cc:648] HybridClock initialized: now 1790608047776898 us; error 47 us; skew 500 ppm
I20260928 15:07:27.777328 28048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:27.779963 28048 webserver.cc:460] Webserver started at http://127.9.12.254:43847/ using document root <none> and password file <none>
I20260928 15:07:27.780766 28048 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:27.780905 28048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:27.781239 28048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:27.784224 28048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "8588574958c74f53b725d631f1693470"
format_stamp: "Formatted at 2026-09-28 15:07:27 on dist-test-slave-dmhs"
server_key: "79340fd4737d6e4250cf9d837fe1fbe7"
server_key_iv: "8704f48efbde75f187280d0fbff94ba5"
server_key_version: "encryptionkey@0"
I20260928 15:07:27.785120 28048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "8588574958c74f53b725d631f1693470"
format_stamp: "Formatted at 2026-09-28 15:07:27 on dist-test-slave-dmhs"
server_key: "79340fd4737d6e4250cf9d837fe1fbe7"
server_key_iv: "8704f48efbde75f187280d0fbff94ba5"
server_key_version: "encryptionkey@0"
I20260928 15:07:27.790776 28048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:07:27.794546 28062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:27.795950 28048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:07:27.796154 28048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "8588574958c74f53b725d631f1693470"
format_stamp: "Formatted at 2026-09-28 15:07:27 on dist-test-slave-dmhs"
server_key: "79340fd4737d6e4250cf9d837fe1fbe7"
server_key_iv: "8704f48efbde75f187280d0fbff94ba5"
server_key_version: "encryptionkey@0"
I20260928 15:07:27.796461 28048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:27.832039 28048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:27.833030 28048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:27.833336 28048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:27.852035 28048 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:42971
I20260928 15:07:27.852043 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:42971 every 8 connection(s)
I20260928 15:07:27.853444 28048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:27.857316 28114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:27.857725 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28048
I20260928 15:07:27.857903 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:27.858381 9267 external_mini_cluster.cc:1428] Setting key 531e25fe595744687ae5b7a955cbd1cd
I20260928 15:07:27.869877 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470: Bootstrap starting.
I20260928 15:07:27.873276 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:27.860342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.9.12.254:42971)
0928 15:07:27.861379 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:27.861390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:27.861446 (+ 56us) server_negotiation.cc:407] Connection header received
0928 15:07:27.863666 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:27.863690 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:27.864192 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:27.864687 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:27.865270 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.866539 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.868010 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.868752 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.868953 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:27.870217 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:27.870228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:27.870246 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:27.870956 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:27.871069 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:27.871096 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:27.871216 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:27.871445 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 15:07:27.871828 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":359,"threads_started":1}
I20260928 15:07:27.874250 28114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:27.875380 28114 log.cc:824] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:27.878224 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470: No bootstrap required, opened a new log
I20260928 15:07:27.885670 28114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8588574958c74f53b725d631f1693470" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42971 } }
I20260928 15:07:27.886070 28114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:27.886181 28114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8588574958c74f53b725d631f1693470, State: Initialized, Role: FOLLOWER
I20260928 15:07:27.886854 28114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8588574958c74f53b725d631f1693470" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42971 } }
I20260928 15:07:27.887102 28114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:27.887236 28114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:27.887419 28114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:27.889983 28114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8588574958c74f53b725d631f1693470" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42971 } }
I20260928 15:07:27.890514 28114 leader_election.cc:302] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8588574958c74f53b725d631f1693470; no voters:
I20260928 15:07:27.891083 28114 leader_election.cc:288] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:27.891243 28119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:27.891988 28119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [term 1 LEADER]: Becoming Leader. State: Replica: 8588574958c74f53b725d631f1693470, State: Running, Role: LEADER
I20260928 15:07:27.892612 28119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8588574958c74f53b725d631f1693470" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42971 } }
I20260928 15:07:27.893193 28114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:27.895448 28120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8588574958c74f53b725d631f1693470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8588574958c74f53b725d631f1693470" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42971 } } }
I20260928 15:07:27.895859 28120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:27.895978 28121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8588574958c74f53b725d631f1693470. Latest consensus state: current_term: 1 leader_uuid: "8588574958c74f53b725d631f1693470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8588574958c74f53b725d631f1693470" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 42971 } } }
I20260928 15:07:27.896291 28121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:27.900553 28126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:27.905771 28126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:27.914829 28126 catalog_manager.cc:1347] Generated new cluster ID: ea6efdb53ae14e89aaf555ac3e675173
I20260928 15:07:27.914981 28126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:27.955237 28126 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:27.956849 28126 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:27.969699 28126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470: Generated new TSK 0
I20260928 15:07:27.970602 28126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:27.975133 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42971
--builtin_ntp_servers=127.9.12.212:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:28.329845 28138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:28.330194 28138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:28.330369 28138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:28.337857 28138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:28.338028 28138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:28.338218 28138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:07:28.347880 28138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:28.349328 28138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:28.351277 28138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:28.364912 28143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:28.364800 28146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:28.364795 28144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:28.365674 28138 server_base.cc:1034] running on GCE node
I20260928 15:07:28.366329 28138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:28.367389 28138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:28.368636 28138 hybrid_clock.cc:648] HybridClock initialized: now 1790608048368572 us; error 63 us; skew 500 ppm
I20260928 15:07:28.368993 28138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:28.371258 28138 webserver.cc:460] Webserver started at http://127.9.12.193:37913/ using document root <none> and password file <none>
I20260928 15:07:28.372164 28138 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:28.372324 28138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:28.372673 28138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:28.375990 28138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "5975ac0d1b3243968968a1bcf9969a64"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-dmhs"
server_key: "46849fc90f560934b0e83d47fb461d3a"
server_key_iv: "2a22f40a58b10ab5a1d865fa0d727fe2"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.377048 28138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "5975ac0d1b3243968968a1bcf9969a64"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-dmhs"
server_key: "46849fc90f560934b0e83d47fb461d3a"
server_key_iv: "2a22f40a58b10ab5a1d865fa0d727fe2"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.382119 28138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:07:28.386037 28152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.387735 28138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:07:28.387902 28138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "5975ac0d1b3243968968a1bcf9969a64"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-dmhs"
server_key: "46849fc90f560934b0e83d47fb461d3a"
server_key_iv: "2a22f40a58b10ab5a1d865fa0d727fe2"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.388137 28138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:28.411180 28138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:28.412117 28138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:28.412485 28138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:28.413740 28138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:28.415347 28138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:28.415464 28138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.415589 28138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:28.415681 28138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.452167 28138 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:38823
I20260928 15:07:28.452314 28264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:38823 every 8 connection(s)
I20260928 15:07:28.454056 28138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:28.456013 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28138
I20260928 15:07:28.456190 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:28.456846 9267 external_mini_cluster.cc:1428] Setting key 6caeb5e3257c231e9ac2176dd16c3710
I20260928 15:07:28.459939 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42971
--builtin_ntp_servers=127.9.12.212:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:28.470777 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:28.459134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:42713 (local address 127.9.12.254:42971)
0928 15:07:28.459607 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:28.459614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:28.461169 (+ 1555us) server_negotiation.cc:407] Connection header received
0928 15:07:28.463001 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:28.463010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:28.463106 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:28.463237 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:28.464319 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.465558 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.467819 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.468230 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.468419 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:28.469981 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:28.469991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:28.469998 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:28.470026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:28.470089 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:28.470104 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:28.470180 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:28.470433 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:07:28.470514 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":262,"threads_started":1}
I20260928 15:07:28.472319 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:28.459817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42971 (local address 127.9.12.193:42713)
0928 15:07:28.460993 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:28.461047 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:28.462723 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:28.463425 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:28.463433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:28.463528 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:28.464112 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:28.464130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.465724 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.465732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:28.467661 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:28.467674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.468509 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.468519 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:28.468782 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:28.469565 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:28.469571 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:28.469625 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:28.469734 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:28.470205 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:28.470219 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:28.470230 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:07:28.470415 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 15:07:28.470865 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":426,"threads_started":1}
I20260928 15:07:28.473932 28265 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42971
I20260928 15:07:28.474259 28265 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:28.475139 28265 heartbeater.cc:511] Master 127.9.12.254:42971 requested a full tablet report, sending...
I20260928 15:07:28.477252 28079 ts_manager.cc:194] Registered new tserver with Master: 5975ac0d1b3243968968a1bcf9969a64 (127.9.12.193:38823)
I20260928 15:07:28.478724 28079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.193:42713
W20260928 15:07:28.803398 28269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:28.803886 28269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:28.804096 28269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:28.812961 28269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:28.813067 28269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:28.813279 28269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:07:28.823573 28269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:28.825011 28269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:28.827036 28269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:28.840777 28275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:28.840809 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:28.841051 28274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:28.841635 28269 server_base.cc:1034] running on GCE node
I20260928 15:07:28.842286 28269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:28.843230 28269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:28.844444 28269 hybrid_clock.cc:648] HybridClock initialized: now 1790608048844386 us; error 46 us; skew 500 ppm
I20260928 15:07:28.844753 28269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:28.847200 28269 webserver.cc:460] Webserver started at http://127.9.12.194:37441/ using document root <none> and password file <none>
I20260928 15:07:28.847981 28269 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:28.848080 28269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:28.848466 28269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:28.851631 28269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "77596f5d49914b648bd73a3ac7c3e249"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-dmhs"
server_key: "d34ff3237aceee40d54823249cd623d5"
server_key_iv: "8d72f3fad0d4eaca72062f017642ccc7"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.852589 28269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "77596f5d49914b648bd73a3ac7c3e249"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-dmhs"
server_key: "d34ff3237aceee40d54823249cd623d5"
server_key_iv: "8d72f3fad0d4eaca72062f017642ccc7"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.858291 28269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 15:07:28.862468 28283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.863922 28269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:07:28.864094 28269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "77596f5d49914b648bd73a3ac7c3e249"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-dmhs"
server_key: "d34ff3237aceee40d54823249cd623d5"
server_key_iv: "8d72f3fad0d4eaca72062f017642ccc7"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.864353 28269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:28.878671 28269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:28.879518 28269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:28.879815 28269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:28.880836 28269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:28.882309 28269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:28.882403 28269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.882566 28269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:28.882653 28269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.916790 28269 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:38779
I20260928 15:07:28.916863 28395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:38779 every 8 connection(s)
I20260928 15:07:28.918192 28269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:28.918819 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28269
I20260928 15:07:28.919010 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:28.919485 9267 external_mini_cluster.cc:1428] Setting key f965d90950e4c46aff62090eb6fc09ff
I20260928 15:07:28.922701 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42971
--builtin_ntp_servers=127.9.12.212:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:28.932186 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:28.921369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:48377 (local address 127.9.12.254:42971)
0928 15:07:28.921583 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:28.921591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:28.923321 (+ 1730us) server_negotiation.cc:407] Connection header received
0928 15:07:28.925187 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:28.925195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:28.925285 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:28.925397 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:28.926444 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.927625 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.929263 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.929964 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.930108 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:28.931450 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:28.931454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:28.931457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:28.931476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:28.931509 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:28.931521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:28.931593 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:28.931863 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:07:28.931954 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:07:28.933532 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:28.922077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42971 (local address 127.9.12.194:48377)
0928 15:07:28.923141 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:28.923186 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:28.924914 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:28.925600 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:28.925608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:28.925697 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:28.926260 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:28.926277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.927783 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.927786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:28.929101 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:28.929114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:28.930163 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:28.930168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:28.930371 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:28.931071 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:28.931076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:28.931130 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:28.931233 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:28.931626 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:28.931641 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:28.931653 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:07:28.931851 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 15:07:28.932161 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":353,"threads_started":1}
I20260928 15:07:28.935037 28396 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42971
I20260928 15:07:28.935338 28396 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:28.936094 28396 heartbeater.cc:511] Master 127.9.12.254:42971 requested a full tablet report, sending...
I20260928 15:07:28.937670 28079 ts_manager.cc:194] Registered new tserver with Master: 77596f5d49914b648bd73a3ac7c3e249 (127.9.12.194:38779)
I20260928 15:07:28.938896 28079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.194:48377
W20260928 15:07:29.252558 28399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:29.252843 28399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:29.252934 28399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:29.260035 28399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:29.260129 28399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:29.260236 28399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:07:29.271030 28399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:42971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:29.272408 28399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:29.274147 28399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:29.287211 28404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:29.287297 28407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:29.287328 28405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:29.288061 28399 server_base.cc:1034] running on GCE node
I20260928 15:07:29.288714 28399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:29.289669 28399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:29.290921 28399 hybrid_clock.cc:648] HybridClock initialized: now 1790608049290872 us; error 54 us; skew 500 ppm
I20260928 15:07:29.291205 28399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:29.293797 28399 webserver.cc:460] Webserver started at http://127.9.12.195:34167/ using document root <none> and password file <none>
I20260928 15:07:29.294498 28399 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:29.294603 28399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:29.294886 28399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:29.297736 28399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "89c1f8136f8749bbb7990f072e93e4fd"
format_stamp: "Formatted at 2026-09-28 15:07:29 on dist-test-slave-dmhs"
server_key: "40124801fe748d49af4c738296376988"
server_key_iv: "1e2b77cc17d0d46aeb2cd25e4f3a8b61"
server_key_version: "encryptionkey@0"
I20260928 15:07:29.298478 28399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "89c1f8136f8749bbb7990f072e93e4fd"
format_stamp: "Formatted at 2026-09-28 15:07:29 on dist-test-slave-dmhs"
server_key: "40124801fe748d49af4c738296376988"
server_key_iv: "1e2b77cc17d0d46aeb2cd25e4f3a8b61"
server_key_version: "encryptionkey@0"
I20260928 15:07:29.303867 28399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:07:29.307389 28413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:29.308899 28399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:07:29.309039 28399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "89c1f8136f8749bbb7990f072e93e4fd"
format_stamp: "Formatted at 2026-09-28 15:07:29 on dist-test-slave-dmhs"
server_key: "40124801fe748d49af4c738296376988"
server_key_iv: "1e2b77cc17d0d46aeb2cd25e4f3a8b61"
server_key_version: "encryptionkey@0"
I20260928 15:07:29.309223 28399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:29.322271 28399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:29.323091 28399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:29.323395 28399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:29.324369 28399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:29.325796 28399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:29.325933 28399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:29.326071 28399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:29.326164 28399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:29.357276 28399 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:37609
I20260928 15:07:29.357321 28525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:37609 every 8 connection(s)
I20260928 15:07:29.358603 28399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:29.361344 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28399
I20260928 15:07:29.361545 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:29.362098 9267 external_mini_cluster.cc:1428] Setting key 6a38622bd45ea763856659a8bc1d43a2
I20260928 15:07:29.371187 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.361265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:37885 (local address 127.9.12.254:42971)
0928 15:07:29.361502 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.361510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.362982 (+ 1472us) server_negotiation.cc:407] Connection header received
0928 15:07:29.364971 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.364975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.365069 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.365175 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:29.366147 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.366880 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.368540 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.369015 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.369121 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.370486 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:29.370490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:29.370493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:29.370511 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:29.370544 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:29.370556 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:29.370628 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.370876 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.370957 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:07:29.372334 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.361833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:42971 (local address 127.9.12.195:37885)
0928 15:07:29.362825 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:29.362872 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:29.364676 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:29.365364 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:29.365374 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:29.365459 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:29.366008 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.366025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.366997 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.367000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.368421 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.368434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.369249 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.369254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.369471 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.370189 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:29.370192 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:29.370232 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:29.370303 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:29.370671 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:29.370681 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:29.370687 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:29.370855 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 15:07:29.371139 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":315,"threads_started":1}
I20260928 15:07:29.373690 28526 heartbeater.cc:347] Connected to a master server at 127.9.12.254:42971
I20260928 15:07:29.373997 28526 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:29.374702 28526 heartbeater.cc:511] Master 127.9.12.254:42971 requested a full tablet report, sending...
I20260928 15:07:29.376144 28079 ts_manager.cc:194] Registered new tserver with Master: 89c1f8136f8749bbb7990f072e93e4fd (127.9.12.195:37609)
I20260928 15:07:29.376992 28079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.195:37885
I20260928 15:07:29.377529 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:29.387555 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.381739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.9.12.254:42971)
0928 15:07:29.381926 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.381931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.382476 (+ 545us) server_negotiation.cc:407] Connection header received
0928 15:07:29.382596 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.382600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.382674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.382759 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:29.383162 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.383825 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.385327 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.385760 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.385861 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.386912 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:29.386917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:29.386920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:29.386956 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:29.386988 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:29.387000 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:29.387113 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.387273 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.387317 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:07:29.391920 28079 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:29.399140 28079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:29.433918 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.411586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.9.12.194:38779)
0928 15:07:29.423557 (+ 11971us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.423568 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.424532 (+ 964us) server_negotiation.cc:407] Connection header received
0928 15:07:29.425060 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.425073 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.425407 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.425616 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:29.426969 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.428698 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.431306 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.432474 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.432601 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.432882 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.433492 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.433619 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11806,"thread_start_us":11678,"threads_started":1}
I20260928 15:07:29.433951 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.411282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:38779 (local address 127.0.0.1:54560)
0928 15:07:29.424320 (+ 13038us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:29.424373 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:29.424732 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:29.426081 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:29.426091 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:29.426135 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:29.426682 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.426698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.428897 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.428901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.431135 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.431146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.432995 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.433001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.433225 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.433280 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 15:07:29.433439 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 15:07:29.433621 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12668,"thread_start_us":16437,"threads_started":1}
I20260928 15:07:29.438035 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.411706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.9.12.193:38823)
0928 15:07:29.422911 (+ 11205us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.422925 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.428137 (+ 5212us) server_negotiation.cc:407] Connection header received
0928 15:07:29.429404 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.429414 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.429670 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.429834 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:29.430807 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.432366 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.434994 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.436495 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.436628 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.436882 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.437476 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.437662 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10966,"thread_start_us":11235,"threads_started":1}
I20260928 15:07:29.438673 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.413396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:37609 (local address 127.0.0.1:44884)
0928 15:07:29.428505 (+ 15109us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:29.428529 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:29.428682 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:29.429957 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:29.429961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:29.429986 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:29.430451 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.430465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.432609 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.432614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.436030 (+ 3416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.436044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.437858 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.437864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.438040 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.438081 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:07:29.438172 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:07:29.438310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14933,"mutex_wait_us":1,"thread_start_us":14765,"threads_started":1}
I20260928 15:07:29.438784 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.411477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38823 (local address 127.0.0.1:48742)
0928 15:07:29.428002 (+ 16525us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:29.428023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:29.429164 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:29.430135 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:29.430143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:29.430188 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:29.430619 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.430629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.432541 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.432546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.434768 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.434787 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.437010 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.437016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.437180 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.437294 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 15:07:29.438294 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0928 15:07:29.438413 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16380,"mutex_wait_us":81,"spinlock_wait_cycles":1408,"thread_start_us":12299,"threads_started":1}
I20260928 15:07:29.438417 28331 tablet_service.cc:1467] Processing CreateTablet for tablet cbdb93eaf5ba4451b8f158489bf68d93 (DEFAULT_TABLE table=test-table [id=7bebf86b83c94c2285ec4d4c6db32fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:29.439517 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.414226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.9.12.195:37609)
0928 15:07:29.429514 (+ 15288us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.429520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.429547 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:07:29.429617 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.429621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.429777 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.429967 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:29.430646 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.432466 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.436240 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.437657 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.437794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.438060 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.438972 (+ 912us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.439141 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15147,"thread_start_us":14721,"threads_started":1}
I20260928 15:07:29.439860 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdb93eaf5ba4451b8f158489bf68d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:29.442794 28200 tablet_service.cc:1467] Processing CreateTablet for tablet cbdb93eaf5ba4451b8f158489bf68d93 (DEFAULT_TABLE table=test-table [id=7bebf86b83c94c2285ec4d4c6db32fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:29.443224 28461 tablet_service.cc:1467] Processing CreateTablet for tablet cbdb93eaf5ba4451b8f158489bf68d93 (DEFAULT_TABLE table=test-table [id=7bebf86b83c94c2285ec4d4c6db32fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:29.444252 28200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdb93eaf5ba4451b8f158489bf68d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:29.444657 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdb93eaf5ba4451b8f158489bf68d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:29.457828 28544 tablet_bootstrap.cc:492] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Bootstrap starting.
I20260928 15:07:29.462178 28546 tablet_bootstrap.cc:492] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Bootstrap starting.
I20260928 15:07:29.464586 28544 tablet_bootstrap.cc:654] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:29.466279 28544 log.cc:824] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:29.466602 28546 tablet_bootstrap.cc:654] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:29.467875 28546 log.cc:824] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:29.469308 28545 tablet_bootstrap.cc:492] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Bootstrap starting.
I20260928 15:07:29.472050 28546 tablet_bootstrap.cc:492] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: No bootstrap required, opened a new log
I20260928 15:07:29.472390 28546 ts_tablet_manager.cc:1397] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:07:29.472499 28544 tablet_bootstrap.cc:492] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: No bootstrap required, opened a new log
I20260928 15:07:29.472638 28545 tablet_bootstrap.cc:654] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:29.472762 28544 ts_tablet_manager.cc:1397] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260928 15:07:29.473815 28545 log.cc:824] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:29.476790 28545 tablet_bootstrap.cc:492] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: No bootstrap required, opened a new log
I20260928 15:07:29.477043 28545 ts_tablet_manager.cc:1397] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 15:07:29.480549 28546 raft_consensus.cc:348] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.480934 28546 raft_consensus.cc:374] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:29.481072 28546 raft_consensus.cc:729] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c1f8136f8749bbb7990f072e93e4fd, State: Initialized, Role: FOLLOWER
I20260928 15:07:29.481729 28265 heartbeater.cc:503] Master 127.9.12.254:42971 was elected leader, sending a full tablet report...
I20260928 15:07:29.481670 28546 consensus_queue.cc:260] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.482728 28526 heartbeater.cc:503] Master 127.9.12.254:42971 was elected leader, sending a full tablet report...
I20260928 15:07:29.484704 28546 ts_tablet_manager.cc:1428] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 15:07:29.485760 28544 raft_consensus.cc:348] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.486260 28544 raft_consensus.cc:374] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:29.486407 28544 raft_consensus.cc:729] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77596f5d49914b648bd73a3ac7c3e249, State: Initialized, Role: FOLLOWER
I20260928 15:07:29.487061 28544 consensus_queue.cc:260] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.487326 28545 raft_consensus.cc:348] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.487808 28545 raft_consensus.cc:374] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:29.487941 28545 raft_consensus.cc:729] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5975ac0d1b3243968968a1bcf9969a64, State: Initialized, Role: FOLLOWER
I20260928 15:07:29.488245 28396 heartbeater.cc:503] Master 127.9.12.254:42971 was elected leader, sending a full tablet report...
I20260928 15:07:29.488574 28545 consensus_queue.cc:260] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.490188 28544 ts_tablet_manager.cc:1428] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
I20260928 15:07:29.490188 28545 ts_tablet_manager.cc:1428] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260928 15:07:29.610378 28527 tablet.cc:2369] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:29.643074 28552 raft_consensus.cc:482] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:29.643380 28552 raft_consensus.cc:504] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.645263 28552 leader_election.cc:288] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77596f5d49914b648bd73a3ac7c3e249 (127.9.12.194:38779), 89c1f8136f8749bbb7990f072e93e4fd (127.9.12.195:37609)
I20260928 15:07:29.651983 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.645755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:54933 (local address 127.9.12.194:38779)
0928 15:07:29.646006 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.646022 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.646571 (+ 549us) server_negotiation.cc:407] Connection header received
0928 15:07:29.646827 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.646831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.646897 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.646992 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:29.647503 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.648304 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.650247 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.651049 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.651233 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.651307 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.651596 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.651724 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:07:29.652092 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.645842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:35289 (local address 127.9.12.195:37609)
0928 15:07:29.646248 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.646256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.646960 (+ 704us) server_negotiation.cc:407] Connection header received
0928 15:07:29.647167 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.647172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.647229 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.647325 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:29.647865 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.648649 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.650435 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.651172 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.651353 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.651441 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.651728 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.651853 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260928 15:07:29.652117 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.645463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:38779 (local address 127.9.12.193:54933)
0928 15:07:29.646449 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:29.646465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:29.646655 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:29.647026 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:29.647030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:29.647058 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:29.647355 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.647364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.648441 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.648446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.650063 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.650075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.651233 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.651239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.651433 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.651458 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:07:29.651566 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:07:29.651691 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":110,"thread_start_us":649,"threads_started":1}
I20260928 15:07:29.652218 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.645754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:37609 (local address 127.9.12.193:35289)
0928 15:07:29.646840 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:29.646854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:29.647029 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:29.647354 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:29.647358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:29.647382 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:29.647710 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.647718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.648795 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.648802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.650256 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:29.650268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.651361 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.651368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:29.651576 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.651598 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:07:29.651700 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:07:29.651818 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":47,"thread_start_us":693,"threads_started":1}
I20260928 15:07:29.653164 28481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdb93eaf5ba4451b8f158489bf68d93" candidate_uuid: "5975ac0d1b3243968968a1bcf9969a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c1f8136f8749bbb7990f072e93e4fd" is_pre_election: true
I20260928 15:07:29.653268 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdb93eaf5ba4451b8f158489bf68d93" candidate_uuid: "5975ac0d1b3243968968a1bcf9969a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77596f5d49914b648bd73a3ac7c3e249" is_pre_election: true
I20260928 15:07:29.653723 28481 raft_consensus.cc:2454] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5975ac0d1b3243968968a1bcf9969a64 in term 0.
I20260928 15:07:29.653743 28351 raft_consensus.cc:2454] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5975ac0d1b3243968968a1bcf9969a64 in term 0.
I20260928 15:07:29.654386 28156 leader_election.cc:302] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5975ac0d1b3243968968a1bcf9969a64, 77596f5d49914b648bd73a3ac7c3e249; no voters:
I20260928 15:07:29.654773 28552 raft_consensus.cc:2781] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:29.654929 28552 raft_consensus.cc:482] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:29.655052 28552 raft_consensus.cc:3037] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:29.657298 28552 raft_consensus.cc:504] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.657922 28552 leader_election.cc:288] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [CANDIDATE]: Term 1 election: Requested vote from peers 77596f5d49914b648bd73a3ac7c3e249 (127.9.12.194:38779), 89c1f8136f8749bbb7990f072e93e4fd (127.9.12.195:37609)
I20260928 15:07:29.658345 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdb93eaf5ba4451b8f158489bf68d93" candidate_uuid: "5975ac0d1b3243968968a1bcf9969a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77596f5d49914b648bd73a3ac7c3e249"
I20260928 15:07:29.658561 28351 raft_consensus.cc:3037] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:29.658495 28481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdb93eaf5ba4451b8f158489bf68d93" candidate_uuid: "5975ac0d1b3243968968a1bcf9969a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c1f8136f8749bbb7990f072e93e4fd"
I20260928 15:07:29.658740 28481 raft_consensus.cc:3037] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:29.660890 28351 raft_consensus.cc:2454] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5975ac0d1b3243968968a1bcf9969a64 in term 1.
I20260928 15:07:29.660972 28481 raft_consensus.cc:2454] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5975ac0d1b3243968968a1bcf9969a64 in term 1.
I20260928 15:07:29.661346 28156 leader_election.cc:302] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5975ac0d1b3243968968a1bcf9969a64, 77596f5d49914b648bd73a3ac7c3e249; no voters:
I20260928 15:07:29.661685 28552 raft_consensus.cc:2781] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:29.662174 28552 raft_consensus.cc:686] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 1 LEADER]: Becoming Leader. State: Replica: 5975ac0d1b3243968968a1bcf9969a64, State: Running, Role: LEADER
I20260928 15:07:29.662658 28552 consensus_queue.cc:237] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } }
I20260928 15:07:29.667459 28079 catalog_manager.cc:5495] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5975ac0d1b3243968968a1bcf9969a64 (127.9.12.193). New cstate: current_term: 1 leader_uuid: "5975ac0d1b3243968968a1bcf9969a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5975ac0d1b3243968968a1bcf9969a64" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 } health_report { overall_health: UNKNOWN } } }
W20260928 15:07:29.671315 28397 tablet.cc:2369] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:29.708844 28266 tablet.cc:2369] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:29.736641 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.731018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.9.12.193:38823)
0928 15:07:29.731203 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.731208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.731227 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:07:29.731371 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.731374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.731485 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.731619 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:29.731971 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.732765 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.734244 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.734695 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.734810 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.735495 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:29.736009 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:29.736156 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.736376 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.736416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:07:29.744969 28481 raft_consensus.cc:1261] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 1 FOLLOWER]: Refusing update from remote peer 5975ac0d1b3243968968a1bcf9969a64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:29.744982 28351 raft_consensus.cc:1261] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 1 FOLLOWER]: Refusing update from remote peer 5975ac0d1b3243968968a1bcf9969a64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:29.745913 28555 consensus_queue.cc:1035] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77596f5d49914b648bd73a3ac7c3e249" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 38779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:29.746296 28552 consensus_queue.cc:1035] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c1f8136f8749bbb7990f072e93e4fd" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 37609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:29.755225 28558 mvcc.cc:205] Tried to move back new op lower bound from 7334330571745550336 to 7334330571422044160. Current Snapshot: MvccSnapshot[applied={T|T < 7334330571745550336}]
I20260928 15:07:29.759893 28559 mvcc.cc:205] Tried to move back new op lower bound from 7334330571745550336 to 7334330571422044160. Current Snapshot: MvccSnapshot[applied={T|T < 7334330571745550336}]
I20260928 15:07:29.774899 28079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:29.775182 28079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:29.778968 28079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8588574958c74f53b725d631f1693470: Sending DeleteTablet for 3 replicas of tablet cbdb93eaf5ba4451b8f158489bf68d93
I20260928 15:07:29.780289 28200 tablet_service.cc:1514] Processing DeleteTablet for tablet cbdb93eaf5ba4451b8f158489bf68d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:29 UTC) from {username='slave'} at 127.0.0.1:48742
I20260928 15:07:29.780378 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet cbdb93eaf5ba4451b8f158489bf68d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:29 UTC) from {username='slave'} at 127.0.0.1:54560
I20260928 15:07:29.781086 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet cbdb93eaf5ba4451b8f158489bf68d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:29 UTC) from {username='slave'} at 127.0.0.1:44884
I20260928 15:07:29.782490 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28138
I20260928 15:07:29.782557 28569 tablet_replica.cc:330] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: stopping tablet replica
I20260928 15:07:29.783118 28569 raft_consensus.cc:2229] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:29.783735 28569 raft_consensus.cc:2258] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:29.784651 28568 tablet_replica.cc:330] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: stopping tablet replica
I20260928 15:07:29.785152 28568 raft_consensus.cc:2229] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:29.785727 28568 raft_consensus.cc:2258] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:29.786768 28567 tablet_replica.cc:330] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: stopping tablet replica
I20260928 15:07:29.787142 28567 raft_consensus.cc:2229] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:29.787825 28567 raft_consensus.cc:2258] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:29.788781 28568 ts_tablet_manager.cc:1905] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:29.789381 28569 ts_tablet_manager.cc:1905] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:29.795506 28567 ts_tablet_manager.cc:1905] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:29.796128 28568 ts_tablet_manager.cc:1918] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:29.796358 28568 log.cc:1196] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/cbdb93eaf5ba4451b8f158489bf68d93
I20260928 15:07:29.796941 28568 ts_tablet_manager.cc:1939] T cbdb93eaf5ba4451b8f158489bf68d93 P 77596f5d49914b648bd73a3ac7c3e249: Deleting consensus metadata
I20260928 15:07:29.798863 28066 catalog_manager.cc:4841] TS 77596f5d49914b648bd73a3ac7c3e249 (127.9.12.194:38779): tablet cbdb93eaf5ba4451b8f158489bf68d93 (table test-table [id=7bebf86b83c94c2285ec4d4c6db32fbd]) successfully deleted
I20260928 15:07:29.801985 28569 ts_tablet_manager.cc:1918] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:29.802189 28569 log.cc:1196] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/cbdb93eaf5ba4451b8f158489bf68d93
I20260928 15:07:29.802721 28569 ts_tablet_manager.cc:1939] T cbdb93eaf5ba4451b8f158489bf68d93 P 89c1f8136f8749bbb7990f072e93e4fd: Deleting consensus metadata
I20260928 15:07:29.803583 28567 ts_tablet_manager.cc:1918] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:29.803767 28567 log.cc:1196] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/cbdb93eaf5ba4451b8f158489bf68d93
I20260928 15:07:29.804382 28567 ts_tablet_manager.cc:1939] T cbdb93eaf5ba4451b8f158489bf68d93 P 5975ac0d1b3243968968a1bcf9969a64: Deleting consensus metadata
I20260928 15:07:29.805223 28065 catalog_manager.cc:4841] TS 89c1f8136f8749bbb7990f072e93e4fd (127.9.12.195:37609): tablet cbdb93eaf5ba4451b8f158489bf68d93 (table test-table [id=7bebf86b83c94c2285ec4d4c6db32fbd]) successfully deleted
I20260928 15:07:29.806038 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.785572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.9.12.193:38823)
0928 15:07:29.785748 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.785754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.795041 (+ 9287us) server_negotiation.cc:407] Connection header received
0928 15:07:29.795537 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.795544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.795660 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.795735 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:29.796621 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.797901 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.801278 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.802037 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.802142 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.803847 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:29.803858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:29.803876 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:29.804877 (+ 1001us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:29.804937 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:29.804962 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:29.805053 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.805536 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.805648 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:07:29.806195 28064 catalog_manager.cc:4841] TS 5975ac0d1b3243968968a1bcf9969a64 (127.9.12.193:38823): tablet cbdb93eaf5ba4451b8f158489bf68d93 (table test-table [id=7bebf86b83c94c2285ec4d4c6db32fbd]) successfully deleted
I20260928 15:07:29.807291 28260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:29.906482 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28138
I20260928 15:07:29.925655 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28269
I20260928 15:07:29.932758 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:29.926487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.9.12.194:38779)
0928 15:07:29.926656 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:29.926663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:29.926746 (+ 83us) server_negotiation.cc:407] Connection header received
0928 15:07:29.926978 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:29.926983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:29.927056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:29.927171 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:29.927560 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.928372 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.929680 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:29.930115 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:29.930248 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:29.931361 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:29.931369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:29.931382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:29.931950 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:29.932003 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:29.932022 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:29.932143 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:29.932309 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:07:29.932421 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:07:29.933626 28391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:30.024577 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28269
I20260928 15:07:30.041672 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28399
I20260928 15:07:30.049089 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:30.042539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.9.12.195:37609)
0928 15:07:30.042745 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:30.042751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:30.042773 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:07:30.042951 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:30.042956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:30.043055 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:30.043161 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:30.043584 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:30.044367 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:30.045666 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:30.046097 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:30.046251 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:30.047485 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:30.047496 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:30.047515 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:30.048370 (+ 855us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:30.048418 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:30.048437 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:30.048521 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:30.048637 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:07:30.048726 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:07:30.049736 28521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:30.138114 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28399
I20260928 15:07:30.154476 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28048
I20260928 15:07:30.155642 28109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:30.266331 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28048
2026-09-28T15:07:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2942 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:30 dist-test-slave-dmhs krb5kdc[28577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:30 dist-test-slave-dmhs krb5kdc[28577](info): set up 2 sockets
Sep 28 15:07:30 dist-test-slave-dmhs krb5kdc[28577](info): commencing operation
krb5kdc: starting...
W20260928 15:07:31.384918 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:31 dist-test-slave-dmhs krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:07:31.645697 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:36219
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:31.978919 28590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:31.979229 28590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:31.979322 28590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:31.986627 28590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:31.986747 28590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:31.986812 28590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:31.986872 28590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:31.986918 28590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:31.996832 28590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:31.998224 28590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:31.999523 28590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:32.011653 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:32.011716 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:32.011752 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:32.012454 28590 server_base.cc:1034] running on GCE node
I20260928 15:07:32.013322 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:32.014773 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:32.016022 28590 hybrid_clock.cc:648] HybridClock initialized: now 1790608052015963 us; error 48 us; skew 500 ppm
I20260928 15:07:32.016451 28590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:32 dist-test-slave-dmhs krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608052, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:32.019654 28590 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:07:32.021157 28590 webserver.cc:460] Webserver started at http://127.9.12.254:32983/ using document root <none> and password file <none>
I20260928 15:07:32.021809 28590 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:32.021919 28590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:32.022176 28590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:32.025230 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6"
format_stamp: "Formatted at 2026-09-28 15:07:32 on dist-test-slave-dmhs"
server_key: "4c62aed5f0311bd499e80d7750a81388"
server_key_iv: "c6517eafebe355350d6709e4e2e16980"
server_key_version: "encryptionkey@0"
I20260928 15:07:32.025955 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6"
format_stamp: "Formatted at 2026-09-28 15:07:32 on dist-test-slave-dmhs"
server_key: "4c62aed5f0311bd499e80d7750a81388"
server_key_iv: "c6517eafebe355350d6709e4e2e16980"
server_key_version: "encryptionkey@0"
I20260928 15:07:32.031196 28590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:07:32.034569 28605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:32.035873 28590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 15:07:32.036058 28590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6"
format_stamp: "Formatted at 2026-09-28 15:07:32 on dist-test-slave-dmhs"
server_key: "4c62aed5f0311bd499e80d7750a81388"
server_key_iv: "c6517eafebe355350d6709e4e2e16980"
server_key_version: "encryptionkey@0"
I20260928 15:07:32.036269 28590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:32.057449 28590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:32.061661 28590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:32.061964 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:32.080875 28590 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:36219
I20260928 15:07:32.080895 28656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:36219 every 8 connection(s)
I20260928 15:07:32.082293 28590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:32.087852 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:32.091054 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28590
I20260928 15:07:32.091269 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:32.092020 9267 external_mini_cluster.cc:1428] Setting key 664884ffda1b31feb3c2275d7a8239a2
I20260928 15:07:32.107527 28657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6: Bootstrap starting.
Sep 28 15:07:32 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:32.111932 28657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:32.113224 28657 log.cc:824] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:32.116752 28657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6: No bootstrap required, opened a new log
I20260928 15:07:32.123154 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:32.094494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.9.12.254:36219)
0928 15:07:32.095281 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:32.095296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:32.095349 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:07:32.097999 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:32.098034 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:32.098616 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:32.099115 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:32.101194 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:32.103237 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:32.105275 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:32.106362 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:32.106532 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:32.112236 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:32.112248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:32.112289 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:32.112351 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:32.117038 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:32.118054 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:32.118065 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:32.118076 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:32.118324 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:32.118827 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:32.118833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:32.118838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:32.119780 (+ 942us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:32.120158 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:32.120653 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 15:07:32.121409 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
I20260928 15:07:32.126562 28657 raft_consensus.cc:348] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36219 } }
I20260928 15:07:32.127054 28657 raft_consensus.cc:374] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:32.127154 28657 raft_consensus.cc:729] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86fe06450c94334a1ca3bb4f3cb8cf6, State: Initialized, Role: FOLLOWER
I20260928 15:07:32.127889 28657 consensus_queue.cc:260] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36219 } }
I20260928 15:07:32.128221 28657 raft_consensus.cc:388] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:32.128399 28657 raft_consensus.cc:482] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:32.128575 28657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:32.130743 28657 raft_consensus.cc:504] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36219 } }
I20260928 15:07:32.131228 28657 leader_election.cc:302] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d86fe06450c94334a1ca3bb4f3cb8cf6; no voters:
I20260928 15:07:32.131901 28657 leader_election.cc:288] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:32.132094 28662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:32.133402 28662 raft_consensus.cc:686] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [term 1 LEADER]: Becoming Leader. State: Replica: d86fe06450c94334a1ca3bb4f3cb8cf6, State: Running, Role: LEADER
I20260928 15:07:32.134044 28657 sys_catalog.cc:564] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:32.133991 28662 consensus_queue.cc:237] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36219 } }
I20260928 15:07:32.136910 28664 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d86fe06450c94334a1ca3bb4f3cb8cf6. Latest consensus state: current_term: 1 leader_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36219 } } }
I20260928 15:07:32.137226 28664 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:32.137910 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86fe06450c94334a1ca3bb4f3cb8cf6" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 36219 } } }
I20260928 15:07:32.138266 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:32.139815 28670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:32.144767 28670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:32.154110 28670 catalog_manager.cc:1347] Generated new cluster ID: 816a504d429b439e85925ceafd3c9079
I20260928 15:07:32.154228 28670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:32.171470 28670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:32.173379 28670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:32.185830 28670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6: Generated new TSK 0
I20260928 15:07:32.186722 28670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.12.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
Entry for principal kudu/127.9.12.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab.
I20260928 15:07:32.264043 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36219
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:32.613596 28683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:32.613884 28683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:32.614065 28683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:32.622016 28683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:32.622148 28683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:32.622354 28683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:07:32.632799 28683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.193.keytab
--principal=kudu/127.9.12.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:32.634059 28683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:32.635375 28683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:32.648315 28688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:32.649050 28689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:32.650168 28691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:32.651110 28683 server_base.cc:1034] running on GCE node
I20260928 15:07:32.651791 28683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:32.652845 28683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:32.654084 28683 hybrid_clock.cc:648] HybridClock initialized: now 1790608052654040 us; error 64 us; skew 500 ppm
I20260928 15:07:32.654428 28683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:32 dist-test-slave-dmhs krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608052, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:32.657648 28683 init.cc:377] Logged in from keytab as kudu/127.9.12.193@KRBTEST.COM (short username kudu)
I20260928 15:07:32.659104 28683 webserver.cc:460] Webserver started at http://127.9.12.193:38533/ using document root <none> and password file <none>
I20260928 15:07:32.659763 28683 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:32.659868 28683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:32.660122 28683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:32.663024 28683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec"
format_stamp: "Formatted at 2026-09-28 15:07:32 on dist-test-slave-dmhs"
server_key: "d913efd999984e4cc7fc6e17cdda7ed3"
server_key_iv: "f62411661684217c4b96894ead7cdfb8"
server_key_version: "encryptionkey@0"
I20260928 15:07:32.663815 28683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec"
format_stamp: "Formatted at 2026-09-28 15:07:32 on dist-test-slave-dmhs"
server_key: "d913efd999984e4cc7fc6e17cdda7ed3"
server_key_iv: "f62411661684217c4b96894ead7cdfb8"
server_key_version: "encryptionkey@0"
I20260928 15:07:32.668900 28683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 15:07:32.672220 28698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:32.673810 28683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:07:32.673979 28683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec"
format_stamp: "Formatted at 2026-09-28 15:07:32 on dist-test-slave-dmhs"
server_key: "d913efd999984e4cc7fc6e17cdda7ed3"
server_key_iv: "f62411661684217c4b96894ead7cdfb8"
server_key_version: "encryptionkey@0"
I20260928 15:07:32.674182 28683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:32.688743 28683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:32.692641 28683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:32.692929 28683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:32.693919 28683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:32.695183 28683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:32.695252 28683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:32.695421 28683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:32.695475 28683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:32.728020 28683 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:39923
I20260928 15:07:32.728070 28810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:39923 every 8 connection(s)
I20260928 15:07:32.729382 28683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:32.734884 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28683
I20260928 15:07:32.735074 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:32.735666 9267 external_mini_cluster.cc:1428] Setting key f339c5f3b3b26466edd6443de7f054f9
Sep 28 15:07:32 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608052, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.193@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:32.751705 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:32.731955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:53697 (local address 127.9.12.254:36219)
0928 15:07:32.732462 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:32.732467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:32.733240 (+ 773us) server_negotiation.cc:407] Connection header received
0928 15:07:32.735499 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:32.735504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:32.735574 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:32.735668 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:32.737566 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:32.738354 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:32.740998 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:32.741466 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:32.741569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:32.745662 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:32.745667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:32.745670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:32.745704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:32.748457 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:32.749229 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:32.749235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:32.749240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:32.749377 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:32.749850 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:32.749854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:32.749856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:32.750140 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:32.750307 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:32.751056 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 15:07:32.751334 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
I20260928 15:07:32.752810 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:32.732397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36219 (local address 127.9.12.193:53697)
0928 15:07:32.733078 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:32.733121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:32.735191 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:32.735960 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:32.735971 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:32.736721 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:32.737371 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:32.737397 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:32.738513 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:32.738523 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:32.740859 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:32.740876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:32.741702 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:32.741711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:32.742037 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:32.742754 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:32.742787 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:32.745414 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:32.748612 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:32.748622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:32.748639 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:32.749093 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:32.749493 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:32.749498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:32.749502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:32.749707 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:32.750275 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:32.750282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:32.750660 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 15:07:32.750935 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 15:07:32.751508 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260928 15:07:32.754592 28811 heartbeater.cc:347] Connected to a master server at 127.9.12.254:36219
I20260928 15:07:32.754958 28811 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:32.755863 28811 heartbeater.cc:511] Master 127.9.12.254:36219 requested a full tablet report, sending...
I20260928 15:07:32.758431 28622 ts_manager.cc:194] Registered new tserver with Master: b7c1b02ca35e495f9d94bcc58b7e29ec (127.9.12.193:39923)
I20260928 15:07:32.760785 28622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.193@KRBTEST.COM'} at 127.9.12.193:53697
WARNING: no policy specified for kudu/127.9.12.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
Entry for principal kudu/127.9.12.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab.
I20260928 15:07:32.797518 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36219
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:33.125025 28817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:33.125308 28817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:33.125471 28817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:33.132906 28817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:33.133021 28817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:33.133214 28817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:07:33.143306 28817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.194.keytab
--principal=kudu/127.9.12.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:33.144717 28817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:33.146044 28817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:33.159148 28825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:33.159231 28823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:33.159471 28822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:33.160094 28817 server_base.cc:1034] running on GCE node
I20260928 15:07:33.160725 28817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:33.161649 28817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:33.162873 28817 hybrid_clock.cc:648] HybridClock initialized: now 1790608053162794 us; error 66 us; skew 500 ppm
I20260928 15:07:33.163188 28817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:33 dist-test-slave-dmhs krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608053, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:33.166213 28817 init.cc:377] Logged in from keytab as kudu/127.9.12.194@KRBTEST.COM (short username kudu)
I20260928 15:07:33.167680 28817 webserver.cc:460] Webserver started at http://127.9.12.194:41285/ using document root <none> and password file <none>
I20260928 15:07:33.168411 28817 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:33.168558 28817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:33.168865 28817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:33.171835 28817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "349c292cfc3e449a914a3b40a9a9e380"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-dmhs"
server_key: "ef756f8c8d4ff9d791db7e2e787f3a24"
server_key_iv: "22c1ffd578e18b5eedd97461d9c4156a"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.172674 28817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "349c292cfc3e449a914a3b40a9a9e380"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-dmhs"
server_key: "ef756f8c8d4ff9d791db7e2e787f3a24"
server_key_iv: "22c1ffd578e18b5eedd97461d9c4156a"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.177944 28817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:07:33.181581 28832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.182899 28817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:07:33.183085 28817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "349c292cfc3e449a914a3b40a9a9e380"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-dmhs"
server_key: "ef756f8c8d4ff9d791db7e2e787f3a24"
server_key_iv: "22c1ffd578e18b5eedd97461d9c4156a"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.183290 28817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:33.201185 28817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:33.205319 28817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:33.205624 28817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:33.206631 28817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:33.208133 28817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:33.208247 28817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.208448 28817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:33.208513 28817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.242050 28817 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:44267
I20260928 15:07:33.242091 28944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:44267 every 8 connection(s)
I20260928 15:07:33.243351 28817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:33.253525 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28817
I20260928 15:07:33.253782 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:33.254339 9267 external_mini_cluster.cc:1428] Setting key c55f45a6a765d3fdbbf154045255100e
Sep 28 15:07:33 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608053, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.194@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:33.263756 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.245900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:59189 (local address 127.9.12.254:36219)
0928 15:07:33.246117 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.246123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.247407 (+ 1284us) server_negotiation.cc:407] Connection header received
0928 15:07:33.249271 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.249275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.249348 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.249438 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:33.251039 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.251725 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.253570 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.254132 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.254231 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.258075 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:33.258080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:33.258083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:33.258115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:33.260744 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:33.261444 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:33.261451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:33.261455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:33.261588 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:33.262010 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:33.262015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:33.262020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:33.262319 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:33.262475 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.263171 (+ 696us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.263399 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:07:33.264722 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.246476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36219 (local address 127.9.12.194:59189)
0928 15:07:33.247231 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.247269 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.249012 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.249645 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.249655 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.250240 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:33.250867 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.250899 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.251895 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.251898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.253432 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.253447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.254363 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.254372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.254662 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.255458 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:33.255497 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:33.257854 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:33.260880 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:33.260889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:33.260905 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:33.261317 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:33.261698 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:33.261704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:33.261707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:33.261890 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:33.262456 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:33.262463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:33.262800 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.263063 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.263605 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":313,"threads_started":1}
I20260928 15:07:33.266145 28945 heartbeater.cc:347] Connected to a master server at 127.9.12.254:36219
I20260928 15:07:33.266471 28945 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:33.267338 28945 heartbeater.cc:511] Master 127.9.12.254:36219 requested a full tablet report, sending...
I20260928 15:07:33.269038 28622 ts_manager.cc:194] Registered new tserver with Master: 349c292cfc3e449a914a3b40a9a9e380 (127.9.12.194:44267)
I20260928 15:07:33.269872 28622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.194@KRBTEST.COM'} at 127.9.12.194:59189
WARNING: no policy specified for kudu/127.9.12.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
Entry for principal kudu/127.9.12.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab.
I20260928 15:07:33.316105 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36219
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:33.644498 28950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:33.644770 28950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:33.644969 28950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:33.652196 28950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:33.652331 28950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:33.652520 28950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:07:33.662333 28950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.195.keytab
--principal=kudu/127.9.12.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:36219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:33.663637 28950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:33.665010 28950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:33.678480 28958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:33.678521 28955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:33.678591 28956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:33.679073 28950 server_base.cc:1034] running on GCE node
I20260928 15:07:33.679741 28950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:33.680747 28950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:33.681991 28950 hybrid_clock.cc:648] HybridClock initialized: now 1790608053681945 us; error 45 us; skew 500 ppm
I20260928 15:07:33.682318 28950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:33 dist-test-slave-dmhs krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608053, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:33.685570 28950 init.cc:377] Logged in from keytab as kudu/127.9.12.195@KRBTEST.COM (short username kudu)
I20260928 15:07:33.687034 28950 webserver.cc:460] Webserver started at http://127.9.12.195:40251/ using document root <none> and password file <none>
I20260928 15:07:33.687731 28950 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:33.687863 28950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:33.688187 28950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:33.691164 28950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "27ecc63cf6dc4e368ed477f84511a56c"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-dmhs"
server_key: "e83877833ba3724f2bcf207a9c88aa27"
server_key_iv: "b8689c98c6290d1e8615747a8a38f6a2"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.691944 28950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "27ecc63cf6dc4e368ed477f84511a56c"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-dmhs"
server_key: "e83877833ba3724f2bcf207a9c88aa27"
server_key_iv: "b8689c98c6290d1e8615747a8a38f6a2"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.697041 28950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:07:33.700397 28965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.701581 28950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:07:33.701745 28950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "27ecc63cf6dc4e368ed477f84511a56c"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-dmhs"
server_key: "e83877833ba3724f2bcf207a9c88aa27"
server_key_iv: "b8689c98c6290d1e8615747a8a38f6a2"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.701936 28950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:33.720444 28950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:33.724933 28950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:33.725244 28950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:33.726227 28950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:33.727788 28950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:33.727900 28950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.728055 28950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:33.728123 28950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.762351 28950 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:34135
I20260928 15:07:33.762404 29077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:34135 every 8 connection(s)
I20260928 15:07:33.763662 28950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:33.764926 28811 heartbeater.cc:503] Master 127.9.12.254:36219 was elected leader, sending a full tablet report...
I20260928 15:07:33.771085 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 28950
I20260928 15:07:33.771287 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:33.771731 9267 external_mini_cluster.cc:1428] Setting key c2125da91189586501e50a50b6a2800d
Sep 28 15:07:33 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608053, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.195@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:07:33.785722 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.766461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:36773 (local address 127.9.12.254:36219)
0928 15:07:33.768730 (+ 2269us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.768739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.768763 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:07:33.770265 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.770271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.770366 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.770483 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:33.772556 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.773720 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.775583 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.776352 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.776450 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.780031 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:33.780036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:33.780039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:33.780072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:33.782526 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:33.783212 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:33.783215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:33.783218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:33.783343 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:33.783710 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:33.783713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:33.783715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:33.783977 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:33.784087 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.785081 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.785369 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":2028,"threads_started":1}
I20260928 15:07:33.786867 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.767223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:36219 (local address 127.9.12.195:36773)
0928 15:07:33.768089 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.768128 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.769996 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.770773 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.770781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.771596 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:33.772359 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.772388 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.773849 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.773853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.775388 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.775419 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.776541 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.776546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.776757 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.777520 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:33.777548 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:33.779821 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:33.782652 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:33.782678 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:33.782692 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:33.783107 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:33.783441 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:33.783444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:33.783446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:33.783618 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:33.784117 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:33.784124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:33.784572 (+ 448us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.784894 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.785426 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":367,"threads_started":1}
I20260928 15:07:33.788502 29078 heartbeater.cc:347] Connected to a master server at 127.9.12.254:36219
I20260928 15:07:33.788771 29078 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:33.789459 29078 heartbeater.cc:511] Master 127.9.12.254:36219 requested a full tablet report, sending...
I20260928 15:07:33.790831 28622 ts_manager.cc:194] Registered new tserver with Master: 27ecc63cf6dc4e368ed477f84511a56c (127.9.12.195:34135)
I20260928 15:07:33.791662 28622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.12.195@KRBTEST.COM'} at 127.9.12.195:36773
I20260928 15:07:33.799981 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:33.817063 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.804490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.9.12.254:36219)
0928 15:07:33.804659 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.804663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.805181 (+ 518us) server_negotiation.cc:407] Connection header received
0928 15:07:33.805304 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.805308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.805393 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.805478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:33.806994 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.807741 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.809179 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.809738 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.809831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.812178 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:33.812182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:33.812185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:33.812215 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:33.814634 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:33.815249 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:33.815252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:33.815255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:33.815375 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:33.815682 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:33.815685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:33.815687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:33.816004 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:33.816175 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.816501 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.816672 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:07:33.820937 28622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:33.827749 28622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:33.860737 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.840505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.9.12.194:44267)
0928 15:07:33.852134 (+ 11629us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.852148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.852185 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:07:33.852334 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.852347 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.852708 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.852966 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:33.853589 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.855285 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.857883 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.859395 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.859574 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.859792 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.860190 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.860402 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11394,"thread_start_us":12701,"threads_started":1}
I20260928 15:07:33.860737 29092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.840180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:44267 (local address 127.0.0.1:43190)
0928 15:07:33.851073 (+ 10893us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.851105 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.851282 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.852973 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.852981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.853011 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:33.853393 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.853406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.855419 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.855424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.857692 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.857708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.859626 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.859632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.859880 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.859944 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.860108 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.860329 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10693,"mutex_wait_us":26,"thread_start_us":10632,"threads_started":1}
I20260928 15:07:33.861907 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.840731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:34135 (local address 127.0.0.1:58608)
0928 15:07:33.851985 (+ 11254us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.852002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.852149 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.853651 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.853655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.853683 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:33.854055 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.854067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.855873 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.855879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.858178 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.858192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.861087 (+ 2895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.861093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.861264 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.861287 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.861392 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.861530 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11123,"mutex_wait_us":28,"thread_start_us":10005,"threads_started":1}
I20260928 15:07:33.861937 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.841459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.9.12.195:34135)
0928 15:07:33.852783 (+ 11324us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.852789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.852814 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:07:33.852922 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.852930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.853184 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.853382 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:33.854263 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.855730 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.858370 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.860021 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.860166 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.860406 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.861441 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.861597 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11155,"thread_start_us":10992,"threads_started":1}
I20260928 15:07:33.863894 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.840191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.9.12.193:39923)
0928 15:07:33.853902 (+ 13711us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.853907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.853928 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:07:33.853997 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.854002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.854212 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.855122 (+ 910us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:33.855408 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.856984 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.859997 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.862400 (+ 2403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.862602 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.862832 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.863397 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.863581 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13563,"thread_start_us":13391,"threads_started":1}
I20260928 15:07:33.864701 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.839886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:39923 (local address 127.0.0.1:36596)
0928 15:07:33.840895 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.840944 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.841110 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.854443 (+ 13333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.854448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.854472 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:33.854922 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.854935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.857133 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.857139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.859802 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.859819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.862673 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.862680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.862846 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.862876 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.862978 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.863114 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":2,"thread_start_us":804,"threads_started":1}
I20260928 15:07:33.865029 28880 tablet_service.cc:1467] Processing CreateTablet for tablet 6575062a29554e4dabb09fad940ee0e2 (DEFAULT_TABLE table=test-table [id=da180638a6f848c0a15f88b02f6557be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:33.865664 29013 tablet_service.cc:1467] Processing CreateTablet for tablet 6575062a29554e4dabb09fad940ee0e2 (DEFAULT_TABLE table=test-table [id=da180638a6f848c0a15f88b02f6557be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:33.866498 28880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575062a29554e4dabb09fad940ee0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:33.867159 29013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575062a29554e4dabb09fad940ee0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:33.868366 28746 tablet_service.cc:1467] Processing CreateTablet for tablet 6575062a29554e4dabb09fad940ee0e2 (DEFAULT_TABLE table=test-table [id=da180638a6f848c0a15f88b02f6557be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:33.869793 28746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575062a29554e4dabb09fad940ee0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:33.878566 29097 tablet_bootstrap.cc:492] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Bootstrap starting.
I20260928 15:07:33.882014 29097 tablet_bootstrap.cc:654] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:33.883481 29098 tablet_bootstrap.cc:492] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Bootstrap starting.
I20260928 15:07:33.883489 29097 log.cc:824] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:33.885038 29099 tablet_bootstrap.cc:492] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Bootstrap starting.
I20260928 15:07:33.886788 29097 tablet_bootstrap.cc:492] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: No bootstrap required, opened a new log
I20260928 15:07:33.887089 29097 ts_tablet_manager.cc:1397] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260928 15:07:33.887414 29098 tablet_bootstrap.cc:654] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:33.888698 29099 tablet_bootstrap.cc:654] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:33.888727 29098 log.cc:824] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:33.889823 29099 log.cc:824] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:33.893008 29098 tablet_bootstrap.cc:492] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: No bootstrap required, opened a new log
I20260928 15:07:33.893342 29098 ts_tablet_manager.cc:1397] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 15:07:33.895223 29097 raft_consensus.cc:348] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.895586 29097 raft_consensus.cc:374] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:33.895741 29097 raft_consensus.cc:729] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349c292cfc3e449a914a3b40a9a9e380, State: Initialized, Role: FOLLOWER
I20260928 15:07:33.896607 29097 consensus_queue.cc:260] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.897758 28945 heartbeater.cc:503] Master 127.9.12.254:36219 was elected leader, sending a full tablet report...
I20260928 15:07:33.898211 29097 ts_tablet_manager.cc:1428] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 15:07:33.899150 29099 tablet_bootstrap.cc:492] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: No bootstrap required, opened a new log
I20260928 15:07:33.899428 29099 ts_tablet_manager.cc:1397] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260928 15:07:33.904280 29098 raft_consensus.cc:348] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.904739 29098 raft_consensus.cc:374] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:33.904850 29098 raft_consensus.cc:729] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ecc63cf6dc4e368ed477f84511a56c, State: Initialized, Role: FOLLOWER
I20260928 15:07:33.905479 29098 consensus_queue.cc:260] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.906678 29078 heartbeater.cc:503] Master 127.9.12.254:36219 was elected leader, sending a full tablet report...
I20260928 15:07:33.907255 29098 ts_tablet_manager.cc:1428] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 15:07:33.909312 29099 raft_consensus.cc:348] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.909668 29099 raft_consensus.cc:374] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:33.909793 29099 raft_consensus.cc:729] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c1b02ca35e495f9d94bcc58b7e29ec, State: Initialized, Role: FOLLOWER
I20260928 15:07:33.910375 29099 consensus_queue.cc:260] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.911893 29099 ts_tablet_manager.cc:1428] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260928 15:07:33.924713 29105 raft_consensus.cc:482] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:33.924986 29105 raft_consensus.cc:504] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.926550 29105 leader_election.cc:288] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349c292cfc3e449a914a3b40a9a9e380 (127.9.12.194:44267), 27ecc63cf6dc4e368ed477f84511a56c (127.9.12.195:34135)
I20260928 15:07:33.934300 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.927077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51491 (local address 127.9.12.195:34135)
0928 15:07:33.927311 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.927320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.928035 (+ 715us) server_negotiation.cc:407] Connection header received
0928 15:07:33.928245 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.928249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.928364 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.928468 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:33.929057 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.929803 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.931944 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.933133 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.933289 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.933350 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.933893 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.934077 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:07:33.934288 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.926899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:55345 (local address 127.9.12.194:44267)
0928 15:07:33.927123 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.927132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.927886 (+ 754us) server_negotiation.cc:407] Connection header received
0928 15:07:33.928127 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.928131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.928187 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.928306 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:33.928923 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.929642 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.931823 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.933045 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.933258 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.933335 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.933905 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.934034 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:07:33.934506 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.926941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:44267 (local address 127.9.12.193:55345)
0928 15:07:33.927772 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.927788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.927958 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.928324 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.928328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.928351 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:33.928768 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.928778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.929776 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.929795 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.931667 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.931683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.933363 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.933376 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.933666 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.933743 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.933871 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.934076 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":225,"thread_start_us":422,"threads_started":1}
I20260928 15:07:33.934528 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.927031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:34135 (local address 127.9.12.193:51491)
0928 15:07:33.927934 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:33.927948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:33.928097 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:33.928486 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:33.928490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:33.928529 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:33.928909 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.928917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.929939 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.929944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.931740 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:33.931753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.933392 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.933398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:33.933669 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.933758 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 15:07:33.933860 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:07:33.934044 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":127,"thread_start_us":571,"threads_started":1}
I20260928 15:07:33.935541 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575062a29554e4dabb09fad940ee0e2" candidate_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349c292cfc3e449a914a3b40a9a9e380" is_pre_election: true
I20260928 15:07:33.936005 28900 raft_consensus.cc:2454] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7c1b02ca35e495f9d94bcc58b7e29ec in term 0.
I20260928 15:07:33.936690 28699 leader_election.cc:302] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349c292cfc3e449a914a3b40a9a9e380, b7c1b02ca35e495f9d94bcc58b7e29ec; no voters:
I20260928 15:07:33.936784 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575062a29554e4dabb09fad940ee0e2" candidate_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ecc63cf6dc4e368ed477f84511a56c" is_pre_election: true
I20260928 15:07:33.937153 29105 raft_consensus.cc:2781] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:33.937299 29105 raft_consensus.cc:482] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:33.937327 29033 raft_consensus.cc:2454] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7c1b02ca35e495f9d94bcc58b7e29ec in term 0.
I20260928 15:07:33.937423 29105 raft_consensus.cc:3037] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:33.939857 29105 raft_consensus.cc:504] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.940524 29105 leader_election.cc:288] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [CANDIDATE]: Term 1 election: Requested vote from peers 349c292cfc3e449a914a3b40a9a9e380 (127.9.12.194:44267), 27ecc63cf6dc4e368ed477f84511a56c (127.9.12.195:34135)
I20260928 15:07:33.940923 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575062a29554e4dabb09fad940ee0e2" candidate_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349c292cfc3e449a914a3b40a9a9e380"
I20260928 15:07:33.941039 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575062a29554e4dabb09fad940ee0e2" candidate_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ecc63cf6dc4e368ed477f84511a56c"
I20260928 15:07:33.941108 28900 raft_consensus.cc:3037] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:33.941270 29033 raft_consensus.cc:3037] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:33.943305 28900 raft_consensus.cc:2454] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7c1b02ca35e495f9d94bcc58b7e29ec in term 1.
I20260928 15:07:33.943490 29033 raft_consensus.cc:2454] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7c1b02ca35e495f9d94bcc58b7e29ec in term 1.
I20260928 15:07:33.943707 28699 leader_election.cc:302] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349c292cfc3e449a914a3b40a9a9e380, b7c1b02ca35e495f9d94bcc58b7e29ec; no voters:
I20260928 15:07:33.943975 29105 raft_consensus.cc:2781] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:33.944434 29105 raft_consensus.cc:686] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 1 LEADER]: Becoming Leader. State: Replica: b7c1b02ca35e495f9d94bcc58b7e29ec, State: Running, Role: LEADER
I20260928 15:07:33.944866 29105 consensus_queue.cc:237] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } }
I20260928 15:07:33.949275 28622 catalog_manager.cc:5495] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec reported cstate change: term changed from 0 to 1, leader changed from <none> to b7c1b02ca35e495f9d94bcc58b7e29ec (127.9.12.193). New cstate: current_term: 1 leader_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c1b02ca35e495f9d94bcc58b7e29ec" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 39923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 } health_report { overall_health: UNKNOWN } } }
W20260928 15:07:33.981798 28812 tablet.cc:2369] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:33.988050 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.982706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.9.12.193:39923)
0928 15:07:33.982919 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.982924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.983007 (+ 83us) server_negotiation.cc:407] Connection header received
0928 15:07:33.983185 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.983188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.983281 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.983405 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:33.983735 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.984544 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.985906 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:33.986279 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:33.986445 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:33.986932 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:33.987567 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:33.987729 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.987771 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.987847 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 15:07:33.995443 28946 tablet.cc:2369] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:33.996194 28900 raft_consensus.cc:1261] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 1 FOLLOWER]: Refusing update from remote peer b7c1b02ca35e495f9d94bcc58b7e29ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:33.996335 29033 raft_consensus.cc:1261] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 1 FOLLOWER]: Refusing update from remote peer b7c1b02ca35e495f9d94bcc58b7e29ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:33.997184 29108 consensus_queue.cc:1035] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "349c292cfc3e449a914a3b40a9a9e380" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:33.997506 29105 consensus_queue.cc:1035] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ecc63cf6dc4e368ed477f84511a56c" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:34.010926 29112 mvcc.cc:205] Tried to move back new op lower bound from 7334330589158445056 to 7334330588961628160. Current Snapshot: MvccSnapshot[applied={T|T < 7334330589158445056}]
I20260928 15:07:34.015105 29111 mvcc.cc:205] Tried to move back new op lower bound from 7334330589158445056 to 7334330588961628160. Current Snapshot: MvccSnapshot[applied={T|T < 7334330589158445056 or (T in {7334330589158445056})}]
I20260928 15:07:34.023592 28621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:34.023944 28621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:34.027882 28621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d86fe06450c94334a1ca3bb4f3cb8cf6: Sending DeleteTablet for 3 replicas of tablet 6575062a29554e4dabb09fad940ee0e2
I20260928 15:07:34.029145 28880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6575062a29554e4dabb09fad940ee0e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:34 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:43190
I20260928 15:07:34.029148 28746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6575062a29554e4dabb09fad940ee0e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:34 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:36596
I20260928 15:07:34.029263 29013 tablet_service.cc:1514] Processing DeleteTablet for tablet 6575062a29554e4dabb09fad940ee0e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:34 UTC) from {username='kudu', principal='kudu/127.9.12.254@KRBTEST.COM'} at 127.0.0.1:58608
I20260928 15:07:34.033569 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28683
I20260928 15:07:34.041838 29120 tablet_replica.cc:330] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: stopping tablet replica
I20260928 15:07:34.042114 29121 tablet_replica.cc:330] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: stopping tablet replica
I20260928 15:07:34.042532 29121 raft_consensus.cc:2229] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:34.042528 29120 raft_consensus.cc:2229] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:34.042851 29122 tablet_replica.cc:330] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: stopping tablet replica
I20260928 15:07:34.043187 29120 raft_consensus.cc:2258] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:34.043264 29122 raft_consensus.cc:2229] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:34.043335 29121 raft_consensus.cc:2258] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:34.045706 29121 ts_tablet_manager.cc:1905] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:34.045851 29120 ts_tablet_manager.cc:1905] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:34.048477 29122 raft_consensus.cc:2258] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:34.053227 29120 ts_tablet_manager.cc:1918] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:34.053423 29120 log.cc:1196] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/6575062a29554e4dabb09fad940ee0e2
I20260928 15:07:34.053689 29121 ts_tablet_manager.cc:1918] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:34.053870 29121 log.cc:1196] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-0/wal/wals/6575062a29554e4dabb09fad940ee0e2
I20260928 15:07:34.054154 29120 ts_tablet_manager.cc:1939] T 6575062a29554e4dabb09fad940ee0e2 P 349c292cfc3e449a914a3b40a9a9e380: Deleting consensus metadata
I20260928 15:07:34.054415 29121 ts_tablet_manager.cc:1939] T 6575062a29554e4dabb09fad940ee0e2 P b7c1b02ca35e495f9d94bcc58b7e29ec: Deleting consensus metadata
I20260928 15:07:34.056197 28606 catalog_manager.cc:4841] TS 349c292cfc3e449a914a3b40a9a9e380 (127.9.12.194:44267): tablet 6575062a29554e4dabb09fad940ee0e2 (table test-table [id=da180638a6f848c0a15f88b02f6557be]) successfully deleted
I20260928 15:07:34.056659 28609 catalog_manager.cc:4841] TS b7c1b02ca35e495f9d94bcc58b7e29ec (127.9.12.193:39923): tablet 6575062a29554e4dabb09fad940ee0e2 (table test-table [id=da180638a6f848c0a15f88b02f6557be]) successfully deleted
I20260928 15:07:34.057687 29122 ts_tablet_manager.cc:1905] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:07:34 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.193@KRBTEST.COM
I20260928 15:07:34.064707 29122 ts_tablet_manager.cc:1918] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:34.064874 29122 log.cc:1196] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/6575062a29554e4dabb09fad940ee0e2
I20260928 15:07:34.065534 29122 ts_tablet_manager.cc:1939] T 6575062a29554e4dabb09fad940ee0e2 P 27ecc63cf6dc4e368ed477f84511a56c: Deleting consensus metadata
I20260928 15:07:34.066150 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:34.034958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.9.12.193:39923)
0928 15:07:34.035207 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:34.035216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:34.047842 (+ 12626us) server_negotiation.cc:407] Connection header received
0928 15:07:34.048091 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:34.048097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:34.048189 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:34.048331 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:34.050801 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.052221 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.054406 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.055069 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.055165 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:34.060023 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:34.060035 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:34.060047 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:34.060088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:34.063091 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.063796 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.063803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.063805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.063950 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.064393 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.064405 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.064407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.064751 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:34.065053 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:34.065519 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 15:07:34.065767 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:07:34.066818 28806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:34.067416 28608 catalog_manager.cc:4841] TS 27ecc63cf6dc4e368ed477f84511a56c (127.9.12.195:34135): tablet 6575062a29554e4dabb09fad940ee0e2 (table test-table [id=da180638a6f848c0a15f88b02f6557be]) successfully deleted
I20260928 15:07:34.166159 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28683
I20260928 15:07:34.185562 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28817
Sep 28 15:07:34 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.194@KRBTEST.COM
I20260928 15:07:34.201601 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:34.186521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.9.12.194:44267)
0928 15:07:34.186691 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:34.186696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:34.186712 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:07:34.187154 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:34.187159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:34.187285 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:34.187379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:34.189023 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.189782 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.191128 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.191517 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.191610 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:34.195982 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:34.195996 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:34.196011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:34.196043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:34.198902 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.199645 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.199652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.199654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.199772 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.200127 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.200130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.200132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.200635 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:34.200858 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:34.201174 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:07:34.201334 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:07:34.202342 28940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:34.291822 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28817
I20260928 15:07:34.311479 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28950
Sep 28 15:07:34 dist-test-slave-dmhs krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.195@KRBTEST.COM
I20260928 15:07:34.326809 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:34.312367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.9.12.195:34135)
0928 15:07:34.312564 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:34.312568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:34.312587 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:07:34.312704 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:34.312708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:34.312784 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:34.312863 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:34.314186 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.314923 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.316166 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.316693 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.316784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:34.321254 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:34.321270 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:34.321287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:34.321320 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:34.324073 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.324902 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.324908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.324910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.325052 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.325439 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.325442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.325444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.325773 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:34.325984 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:34.326268 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:07:34.326457 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:07:34.327440 29073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:34.416673 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28950
I20260928 15:07:34.433483 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 28590
I20260928 15:07:34.434672 28652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:34.550161 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 28590
2026-09-28T15:07:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4283 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T15:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:34Z Disabled control of system clock
I20260928 15:07:34.629259 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43239
--webserver_interface=127.9.12.254
--webserver_port=0
--builtin_ntp_servers=127.9.12.212:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:34.966408 29131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:34.966713 29131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:34.966815 29131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:34.974253 29131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:34.974370 29131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:34.974418 29131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:34.974478 29131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:34.974526 29131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:34.983876 29131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:43239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:34.985167 29131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:34.986892 29131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:34.998374 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:34.998411 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:34.998337 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:34.999686 29131 server_base.cc:1034] running on GCE node
I20260928 15:07:35.000542 29131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:35.001922 29131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:35.003124 29131 hybrid_clock.cc:648] HybridClock initialized: now 1790608055003072 us; error 41 us; skew 500 ppm
I20260928 15:07:35.003487 29131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:35.005867 29131 webserver.cc:460] Webserver started at http://127.9.12.254:46299/ using document root <none> and password file <none>
I20260928 15:07:35.006567 29131 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:35.006691 29131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:35.007062 29131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:35.010097 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "050660a30a784dbea500f8f3440c4425"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-dmhs"
server_key: "3f291d12b84353a90eb6eed47931eea3"
server_key_iv: "85e67fab89b4f84d2a5f085a24a9f822"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.010973 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "050660a30a784dbea500f8f3440c4425"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-dmhs"
server_key: "3f291d12b84353a90eb6eed47931eea3"
server_key_iv: "85e67fab89b4f84d2a5f085a24a9f822"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.016147 29131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:07:35.019577 29145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:35.020984 29131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:35.021181 29131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "050660a30a784dbea500f8f3440c4425"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-dmhs"
server_key: "3f291d12b84353a90eb6eed47931eea3"
server_key_iv: "85e67fab89b4f84d2a5f085a24a9f822"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.021412 29131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:35.041493 29131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:35.042414 29131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:35.042723 29131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:35.061031 29131 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:43239
I20260928 15:07:35.061053 29196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:43239 every 8 connection(s)
I20260928 15:07:35.062378 29131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:35.064627 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29131
I20260928 15:07:35.064821 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:35.065276 9267 external_mini_cluster.cc:1428] Setting key 1503373892697983249cc4fe531bc489
I20260928 15:07:35.066497 29197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:35.078326 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:35.067150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.9.12.254:43239)
0928 15:07:35.068131 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:35.068149 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:35.068194 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:07:35.069826 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:35.069846 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:35.070309 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:35.070712 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:35.071256 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:35.072607 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:35.073934 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:35.074598 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:35.074797 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:35.076038 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:35.076049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:35.076062 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:35.076671 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:35.076749 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:35.076770 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:35.076902 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:35.077113 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 15:07:35.077417 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":347,"threads_started":1}
I20260928 15:07:35.078419 29197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425: Bootstrap starting.
I20260928 15:07:35.081802 29197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:35.082768 29197 log.cc:824] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:35.085422 29197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425: No bootstrap required, opened a new log
I20260928 15:07:35.092918 29197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050660a30a784dbea500f8f3440c4425" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43239 } }
I20260928 15:07:35.093236 29197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:35.093307 29197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050660a30a784dbea500f8f3440c4425, State: Initialized, Role: FOLLOWER
I20260928 15:07:35.093765 29197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050660a30a784dbea500f8f3440c4425" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43239 } }
I20260928 15:07:35.093961 29197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:35.094058 29197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:35.094178 29197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:35.096146 29197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050660a30a784dbea500f8f3440c4425" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43239 } }
I20260928 15:07:35.096640 29197 leader_election.cc:302] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050660a30a784dbea500f8f3440c4425; no voters:
I20260928 15:07:35.097173 29197 leader_election.cc:288] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:35.097342 29202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:35.098073 29202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [term 1 LEADER]: Becoming Leader. State: Replica: 050660a30a784dbea500f8f3440c4425, State: Running, Role: LEADER
I20260928 15:07:35.098644 29202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050660a30a784dbea500f8f3440c4425" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43239 } }
I20260928 15:07:35.099267 29197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:35.101778 29203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "050660a30a784dbea500f8f3440c4425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050660a30a784dbea500f8f3440c4425" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43239 } } }
I20260928 15:07:35.102269 29203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:35.102834 29204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 050660a30a784dbea500f8f3440c4425. Latest consensus state: current_term: 1 leader_uuid: "050660a30a784dbea500f8f3440c4425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050660a30a784dbea500f8f3440c4425" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43239 } } }
I20260928 15:07:35.103219 29204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:35.105489 29209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:35.112035 29209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:35.120916 29209 catalog_manager.cc:1347] Generated new cluster ID: ab64797e59264758ab1abf99f12be974
I20260928 15:07:35.121045 29209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:35.140193 29209 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:35.142099 29209 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:35.151306 29209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425: Generated new TSK 0
I20260928 15:07:35.152137 29209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:35.163970 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.193:0
--local_ip_for_outbound_sockets=127.9.12.193
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43239
--builtin_ntp_servers=127.9.12.212:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:35.523553 29221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:35.523880 29221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:35.524050 29221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:35.531704 29221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:35.531850 29221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:35.532038 29221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.193
I20260928 15:07:35.541487 29221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.12.193
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.193
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:35.542873 29221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:35.544670 29221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:35.557971 29226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:35.557940 29229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:35.558027 29227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:35.558732 29221 server_base.cc:1034] running on GCE node
I20260928 15:07:35.559378 29221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:35.560372 29221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:35.561597 29221 hybrid_clock.cc:648] HybridClock initialized: now 1790608055561538 us; error 55 us; skew 500 ppm
I20260928 15:07:35.561918 29221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:35.564139 29221 webserver.cc:460] Webserver started at http://127.9.12.193:44627/ using document root <none> and password file <none>
I20260928 15:07:35.564867 29221 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:35.565004 29221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:35.565331 29221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:35.568166 29221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data/instance:
uuid: "485a651d3afb473ebc747e44c4367ca8"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-dmhs"
server_key: "104cff3cc5597ee7d7763604bed960a1"
server_key_iv: "df25d6d3eefe23fd5f5fea485ac49e37"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.569024 29221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance:
uuid: "485a651d3afb473ebc747e44c4367ca8"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-dmhs"
server_key: "104cff3cc5597ee7d7763604bed960a1"
server_key_iv: "df25d6d3eefe23fd5f5fea485ac49e37"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.573962 29221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:07:35.577366 29235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:35.578823 29221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:07:35.579000 29221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal
uuid: "485a651d3afb473ebc747e44c4367ca8"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-dmhs"
server_key: "104cff3cc5597ee7d7763604bed960a1"
server_key_iv: "df25d6d3eefe23fd5f5fea485ac49e37"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.579216 29221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:35.600477 29221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:35.601343 29221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:35.601653 29221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:35.602584 29221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:35.604100 29221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:35.604229 29221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:35.604386 29221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:35.604480 29221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:35.636780 29221 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.193:38243
I20260928 15:07:35.636864 29347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.193:38243 every 8 connection(s)
I20260928 15:07:35.638077 29221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:35.644454 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29221
I20260928 15:07:35.644642 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:35.645090 9267 external_mini_cluster.cc:1428] Setting key 3a66d516ef7354cdfd5c1c2e94f34a8b
I20260928 15:07:35.647861 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.194:0
--local_ip_for_outbound_sockets=127.9.12.194
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43239
--builtin_ntp_servers=127.9.12.212:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:35.651372 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:35.640721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.193:51961 (local address 127.9.12.254:43239)
0928 15:07:35.641238 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:35.641242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:35.642282 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 15:07:35.644130 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:35.644136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:35.644231 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:35.644405 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:35.645628 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:35.646519 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:35.648192 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:35.648990 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:35.649147 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:35.650461 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:35.650470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:35.650478 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:35.650503 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:35.650557 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:35.650575 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:35.650669 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:35.650911 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 15:07:35.651024 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":306,"threads_started":1}
I20260928 15:07:35.652531 29351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:35.641221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43239 (local address 127.9.12.193:51961)
0928 15:07:35.642116 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:35.642158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:35.643836 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:35.644654 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:35.644675 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:35.644814 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:35.645458 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:35.645478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:35.646698 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:35.646702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:35.647965 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:35.647978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:35.649202 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:35.649207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:35.649420 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:35.650134 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:35.650137 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:35.650178 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:35.650261 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:35.650678 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:35.650687 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:35.650692 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:35.650882 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 15:07:35.651306 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":274,"threads_started":1}
I20260928 15:07:35.655087 29348 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43239
I20260928 15:07:35.655427 29348 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:35.656373 29348 heartbeater.cc:511] Master 127.9.12.254:43239 requested a full tablet report, sending...
I20260928 15:07:35.658574 29162 ts_manager.cc:194] Registered new tserver with Master: 485a651d3afb473ebc747e44c4367ca8 (127.9.12.193:38243)
I20260928 15:07:35.660015 29162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.193:51961
W20260928 15:07:35.982595 29352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:35.982913 29352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:35.983069 29352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:35.990477 29352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:35.990636 29352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:35.990820 29352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.194
I20260928 15:07:35.999971 29352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.12.194
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.194
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:36.001368 29352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:36.003237 29352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:36.016600 29357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:36.016670 29358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:36.016670 29360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:36.017819 29352 server_base.cc:1034] running on GCE node
I20260928 15:07:36.018527 29352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:36.019480 29352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:36.020720 29352 hybrid_clock.cc:648] HybridClock initialized: now 1790608056020672 us; error 60 us; skew 500 ppm
I20260928 15:07:36.021070 29352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:36.023211 29352 webserver.cc:460] Webserver started at http://127.9.12.194:39973/ using document root <none> and password file <none>
I20260928 15:07:36.023964 29352 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:36.024135 29352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:36.024544 29352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:36.027503 29352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data/instance:
uuid: "500ea281f2ce4034960472c2bb1b2892"
format_stamp: "Formatted at 2026-09-28 15:07:36 on dist-test-slave-dmhs"
server_key: "1816abf281d7392ffe4826077c4f2916"
server_key_iv: "c8f91a2500e52a68b87f2552f182d5ca"
server_key_version: "encryptionkey@0"
I20260928 15:07:36.028349 29352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance:
uuid: "500ea281f2ce4034960472c2bb1b2892"
format_stamp: "Formatted at 2026-09-28 15:07:36 on dist-test-slave-dmhs"
server_key: "1816abf281d7392ffe4826077c4f2916"
server_key_iv: "c8f91a2500e52a68b87f2552f182d5ca"
server_key_version: "encryptionkey@0"
I20260928 15:07:36.033241 29352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 15:07:36.036597 29366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:36.037881 29352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:07:36.038055 29352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal
uuid: "500ea281f2ce4034960472c2bb1b2892"
format_stamp: "Formatted at 2026-09-28 15:07:36 on dist-test-slave-dmhs"
server_key: "1816abf281d7392ffe4826077c4f2916"
server_key_iv: "c8f91a2500e52a68b87f2552f182d5ca"
server_key_version: "encryptionkey@0"
I20260928 15:07:36.038275 29352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:36.061563 29352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:36.062400 29352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:36.062729 29352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:36.063714 29352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:36.065343 29352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:36.065477 29352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:36.065601 29352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:36.065689 29352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:36.099432 29352 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.194:46351
I20260928 15:07:36.099519 29478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.194:46351 every 8 connection(s)
I20260928 15:07:36.100808 29352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:36.107685 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29352
I20260928 15:07:36.107873 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:36.108497 9267 external_mini_cluster.cc:1428] Setting key 323c81d8abfd1305d4620c2d5665033c
I20260928 15:07:36.111435 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.12.195:0
--local_ip_for_outbound_sockets=127.9.12.195
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43239
--builtin_ntp_servers=127.9.12.212:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:36.114259 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.103348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.194:43817 (local address 127.9.12.254:43239)
0928 15:07:36.103550 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.103554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.104971 (+ 1417us) server_negotiation.cc:407] Connection header received
0928 15:07:36.106831 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.106835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.106902 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.106986 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.108447 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.109270 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.111256 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.111734 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.111831 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.113544 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.113549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.113552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.113570 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.113603 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.113615 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.113684 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.113954 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.114032 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:07:36.115731 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.103850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43239 (local address 127.9.12.194:43817)
0928 15:07:36.104815 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.104850 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.106568 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.107243 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.107256 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.107385 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:36.108243 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.108296 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.109421 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.109426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.111109 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.111127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.111962 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.111971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.112346 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.113185 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:36.113190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:36.113245 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:36.113353 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:36.113713 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:36.113728 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:36.113738 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.113925 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.114357 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":279,"threads_started":1}
I20260928 15:07:36.117163 29479 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43239
I20260928 15:07:36.117493 29479 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:36.118397 29479 heartbeater.cc:511] Master 127.9.12.254:43239 requested a full tablet report, sending...
I20260928 15:07:36.119870 29162 ts_manager.cc:194] Registered new tserver with Master: 500ea281f2ce4034960472c2bb1b2892 (127.9.12.194:46351)
I20260928 15:07:36.120720 29162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.194:43817
W20260928 15:07:36.442732 29482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:36.443012 29482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:36.443168 29482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:36.453766 29482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:36.453868 29482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:36.454063 29482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.12.195
I20260928 15:07:36.463238 29482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.12.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.12.195
--webserver_port=0
--tserver_master_addrs=127.9.12.254:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.12.195
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:36.464608 29482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:36.466408 29482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:36.479712 29487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:36.479748 29490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:36.479676 29488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:36.480873 29482 server_base.cc:1034] running on GCE node
I20260928 15:07:36.481554 29482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:36.482649 29482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:36.483918 29482 hybrid_clock.cc:648] HybridClock initialized: now 1790608056483847 us; error 55 us; skew 500 ppm
I20260928 15:07:36.484345 29482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:07:36.486613 29482 webserver.cc:460] Webserver started at http://127.9.12.195:43053/ using document root <none> and password file <none>
I20260928 15:07:36.487324 29482 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:36.487461 29482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:36.487803 29482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:36.490762 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data/instance:
uuid: "9b5b4cc19535491fa3c2beb9bc4e8000"
format_stamp: "Formatted at 2026-09-28 15:07:36 on dist-test-slave-dmhs"
server_key: "89a664951b3b9167f48658d3249023b4"
server_key_iv: "a18dfbf3000665b6cb93e705a756809a"
server_key_version: "encryptionkey@0"
I20260928 15:07:36.491575 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance:
uuid: "9b5b4cc19535491fa3c2beb9bc4e8000"
format_stamp: "Formatted at 2026-09-28 15:07:36 on dist-test-slave-dmhs"
server_key: "89a664951b3b9167f48658d3249023b4"
server_key_iv: "a18dfbf3000665b6cb93e705a756809a"
server_key_version: "encryptionkey@0"
I20260928 15:07:36.496610 29482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:07:36.500212 29496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:36.501580 29482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:07:36.501752 29482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal
uuid: "9b5b4cc19535491fa3c2beb9bc4e8000"
format_stamp: "Formatted at 2026-09-28 15:07:36 on dist-test-slave-dmhs"
server_key: "89a664951b3b9167f48658d3249023b4"
server_key_iv: "a18dfbf3000665b6cb93e705a756809a"
server_key_version: "encryptionkey@0"
I20260928 15:07:36.501981 29482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:36.514230 29482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:36.515033 29482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:36.515331 29482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:36.516358 29482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:36.517829 29482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:36.517926 29482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:36.518069 29482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:36.518162 29482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:36.551218 29482 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.195:44221
I20260928 15:07:36.551263 29608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.195:44221 every 8 connection(s)
I20260928 15:07:36.552697 29482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:36.560200 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29482
I20260928 15:07:36.560421 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:36.560921 9267 external_mini_cluster.cc:1428] Setting key a38c4ebf3111bb4ddeac72f90eba099e
I20260928 15:07:36.565578 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.555261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:34247 (local address 127.9.12.254:43239)
0928 15:07:36.555472 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.555476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.556929 (+ 1453us) server_negotiation.cc:407] Connection header received
0928 15:07:36.558689 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.558693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.558771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.558863 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.559909 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.560757 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.562806 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.563280 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.563377 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.564870 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.564874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.564878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.564896 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.564927 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.564940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.565009 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.565203 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.565282 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:07:36.566530 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.555815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:43239 (local address 127.9.12.195:34247)
0928 15:07:36.556766 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.556807 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.558405 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.559069 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.559076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.559171 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:36.559734 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.559751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.560926 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.560932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.562649 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.562667 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.563500 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.563508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.563811 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.564573 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:36.564576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:36.564619 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:36.564696 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:36.565023 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:36.565030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:36.565036 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.565180 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.565491 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":261,"threads_started":1}
I20260928 15:07:36.567831 29609 heartbeater.cc:347] Connected to a master server at 127.9.12.254:43239
I20260928 15:07:36.568157 29609 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:36.568897 29609 heartbeater.cc:511] Master 127.9.12.254:43239 requested a full tablet report, sending...
I20260928 15:07:36.570319 29162 ts_manager.cc:194] Registered new tserver with Master: 9b5b4cc19535491fa3c2beb9bc4e8000 (127.9.12.195:44221)
I20260928 15:07:36.571152 29162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.12.195:34247
I20260928 15:07:36.576323 9267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:36.586212 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.580463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.9.12.254:43239)
0928 15:07:36.580636 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.580640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.581096 (+ 456us) server_negotiation.cc:407] Connection header received
0928 15:07:36.581242 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.581245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.581313 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.581400 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.581841 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.582531 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.583939 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.584380 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.584481 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.585587 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.585592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.585595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.585619 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.585659 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.585673 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.585764 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.585905 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.585997 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:07:36.590077 29162 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:36.596838 29162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:36.645665 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.609316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.195:44221 (local address 127.0.0.1:33202)
0928 15:07:36.626252 (+ 16936us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.626303 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.626580 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.637823 (+ 11243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.637827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.637870 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:36.638281 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.638295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.640441 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.640446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.642922 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.642936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.644685 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.644690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.644865 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.644928 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.645099 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.645292 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15666,"mutex_wait_us":29,"thread_start_us":15797,"threads_started":1}
I20260928 15:07:36.645658 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.609634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.9.12.195:44221)
0928 15:07:36.637372 (+ 27738us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.637378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.637411 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:07:36.637485 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.637490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.637641 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.637839 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:36.638523 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.640284 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.643145 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.644494 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.644653 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.644943 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.645155 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.645312 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27586,"thread_start_us":27432,"threads_started":1}
I20260928 15:07:36.646539 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.610082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:46351 (local address 127.0.0.1:42248)
0928 15:07:36.638423 (+ 28341us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.638441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.638546 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.639635 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.639640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.639663 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:36.640042 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.640066 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.641972 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.641978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.644525 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.644539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.645796 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.645802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.645953 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.645985 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.646061 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.646184 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28191,"mutex_wait_us":279,"thread_start_us":28553,"threads_started":1}
I20260928 15:07:36.647096 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.610725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.9.12.194:46351)
0928 15:07:36.635521 (+ 24796us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.635529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.638865 (+ 3336us) server_negotiation.cc:407] Connection header received
0928 15:07:36.638971 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.638979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.639217 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.639384 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:36.640285 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.641810 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.644723 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.645600 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.645746 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.646232 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.646293 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.646473 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24627,"thread_start_us":24815,"threads_started":1}
I20260928 15:07:36.649117 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.609505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39154)
0928 15:07:36.635533 (+ 26028us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.635554 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.635708 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.637186 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.637194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.637225 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:36.637604 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.637618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.640867 (+ 3249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.640877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.644413 (+ 3536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.644442 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.648306 (+ 3864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.648313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.648507 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.648537 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.648628 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.648787 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25914,"mutex_wait_us":44,"thread_start_us":26287,"threads_started":1}
I20260928 15:07:36.649288 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.610039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.9.12.193:38243)
0928 15:07:36.634903 (+ 24864us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.634916 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.635708 (+ 792us) server_negotiation.cc:407] Connection header received
0928 15:07:36.636028 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.636037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.636348 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.636579 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:36.637797 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.640702 (+ 2905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.646320 (+ 5618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.647746 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.647882 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.648162 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.648708 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.648903 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24642,"thread_start_us":24896,"threads_started":1}
I20260928 15:07:36.650048 29544 tablet_service.cc:1467] Processing CreateTablet for tablet 7753fc01ca8c470eadd2adc720d63f5e (DEFAULT_TABLE table=test-table [id=e0eaeb884100433a9b8a88f08cc4ecf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:36.650771 29414 tablet_service.cc:1467] Processing CreateTablet for tablet 7753fc01ca8c470eadd2adc720d63f5e (DEFAULT_TABLE table=test-table [id=e0eaeb884100433a9b8a88f08cc4ecf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:36.651623 29544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7753fc01ca8c470eadd2adc720d63f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:36.651863 29414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7753fc01ca8c470eadd2adc720d63f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:36.652844 29283 tablet_service.cc:1467] Processing CreateTablet for tablet 7753fc01ca8c470eadd2adc720d63f5e (DEFAULT_TABLE table=test-table [id=e0eaeb884100433a9b8a88f08cc4ecf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:36.654166 29283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7753fc01ca8c470eadd2adc720d63f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:36.663640 29348 heartbeater.cc:503] Master 127.9.12.254:43239 was elected leader, sending a full tablet report...
I20260928 15:07:36.664536 29627 tablet_bootstrap.cc:492] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Bootstrap starting.
I20260928 15:07:36.668069 29627 tablet_bootstrap.cc:654] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:36.668990 29628 tablet_bootstrap.cc:492] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Bootstrap starting.
I20260928 15:07:36.669443 29627 log.cc:824] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:36.670547 29629 tablet_bootstrap.cc:492] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: Bootstrap starting.
I20260928 15:07:36.672910 29628 tablet_bootstrap.cc:654] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:36.674099 29629 tablet_bootstrap.cc:654] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:36.674162 29627 tablet_bootstrap.cc:492] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: No bootstrap required, opened a new log
I20260928 15:07:36.674525 29627 ts_tablet_manager.cc:1397] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 15:07:36.674619 29628 log.cc:824] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:36.675421 29629 log.cc:824] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:36.677855 29628 tablet_bootstrap.cc:492] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: No bootstrap required, opened a new log
I20260928 15:07:36.678107 29628 ts_tablet_manager.cc:1397] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 15:07:36.679400 29629 tablet_bootstrap.cc:492] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: No bootstrap required, opened a new log
I20260928 15:07:36.679661 29629 ts_tablet_manager.cc:1397] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 15:07:36.685384 29627 raft_consensus.cc:348] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.685843 29627 raft_consensus.cc:374] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:36.685945 29627 raft_consensus.cc:729] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ea281f2ce4034960472c2bb1b2892, State: Initialized, Role: FOLLOWER
I20260928 15:07:36.686590 29627 consensus_queue.cc:260] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.687783 29479 heartbeater.cc:503] Master 127.9.12.254:43239 was elected leader, sending a full tablet report...
I20260928 15:07:36.688036 29629 raft_consensus.cc:348] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.688400 29629 raft_consensus.cc:374] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:36.688529 29629 raft_consensus.cc:729] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485a651d3afb473ebc747e44c4367ca8, State: Initialized, Role: FOLLOWER
I20260928 15:07:36.688582 29628 raft_consensus.cc:348] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.689000 29628 raft_consensus.cc:374] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:36.689101 29628 raft_consensus.cc:729] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5b4cc19535491fa3c2beb9bc4e8000, State: Initialized, Role: FOLLOWER
I20260928 15:07:36.689148 29629 consensus_queue.cc:260] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.689724 29628 consensus_queue.cc:260] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.689971 29627 ts_tablet_manager.cc:1428] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 15:07:36.691561 29629 ts_tablet_manager.cc:1428] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260928 15:07:36.693799 29609 heartbeater.cc:503] Master 127.9.12.254:43239 was elected leader, sending a full tablet report...
I20260928 15:07:36.694440 29628 ts_tablet_manager.cc:1428] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 15:07:36.706442 29635 raft_consensus.cc:482] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:36.706682 29635 raft_consensus.cc:504] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.708416 29635 leader_election.cc:288] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 485a651d3afb473ebc747e44c4367ca8 (127.9.12.193:38243), 500ea281f2ce4034960472c2bb1b2892 (127.9.12.194:46351)
I20260928 15:07:36.714758 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.708961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:52379 (local address 127.9.12.194:46351)
0928 15:07:36.709194 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.709202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.709572 (+ 370us) server_negotiation.cc:407] Connection header received
0928 15:07:36.709717 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.709721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.709786 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.709854 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:36.710549 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.711192 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.712980 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.713837 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.714007 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.714066 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.714380 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.714515 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:07:36.714780 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.708598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.9.12.195:46687)
0928 15:07:36.709181 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.709207 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.709351 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.710014 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.710018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.710064 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:36.710256 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.710264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.711246 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.711254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.712701 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.712712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.714112 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.714117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.714281 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.714322 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.714424 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.714536 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":25}
I20260928 15:07:36.715024 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.708711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.194:46351 (local address 127.9.12.195:52379)
0928 15:07:36.709466 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.709483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:36.709608 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:36.709899 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:36.709902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:36.709922 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:36.710401 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.710412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.711336 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.711341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.712802 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:36.712813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.714023 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.714028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:36.714201 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.714241 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:07:36.714333 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:07:36.714479 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":16,"spinlock_wait_cycles":16640,"thread_start_us":328,"threads_started":1}
I20260928 15:07:36.715046 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.708732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.12.195:46687 (local address 127.9.12.193:38243)
0928 15:07:36.709007 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.709015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.709298 (+ 283us) server_negotiation.cc:407] Connection header received
0928 15:07:36.709493 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.709502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.709599 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.709693 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:36.710407 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.711122 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.712847 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.713816 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.713976 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.714045 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.714680 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.714811 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 15:07:36.715965 29303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7753fc01ca8c470eadd2adc720d63f5e" candidate_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485a651d3afb473ebc747e44c4367ca8" is_pre_election: true
I20260928 15:07:36.715983 29434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7753fc01ca8c470eadd2adc720d63f5e" candidate_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ea281f2ce4034960472c2bb1b2892" is_pre_election: true
I20260928 15:07:36.716478 29303 raft_consensus.cc:2454] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5b4cc19535491fa3c2beb9bc4e8000 in term 0.
I20260928 15:07:36.716493 29434 raft_consensus.cc:2454] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5b4cc19535491fa3c2beb9bc4e8000 in term 0.
I20260928 15:07:36.717144 29500 leader_election.cc:302] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 485a651d3afb473ebc747e44c4367ca8, 500ea281f2ce4034960472c2bb1b2892, 9b5b4cc19535491fa3c2beb9bc4e8000; no voters:
I20260928 15:07:36.717510 29635 raft_consensus.cc:2781] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:36.717621 29635 raft_consensus.cc:482] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:36.717765 29635 raft_consensus.cc:3037] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:36.719942 29635 raft_consensus.cc:504] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.720570 29635 leader_election.cc:288] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [CANDIDATE]: Term 1 election: Requested vote from peers 485a651d3afb473ebc747e44c4367ca8 (127.9.12.193:38243), 500ea281f2ce4034960472c2bb1b2892 (127.9.12.194:46351)
I20260928 15:07:36.721100 29434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7753fc01ca8c470eadd2adc720d63f5e" candidate_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ea281f2ce4034960472c2bb1b2892"
I20260928 15:07:36.721100 29303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7753fc01ca8c470eadd2adc720d63f5e" candidate_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485a651d3afb473ebc747e44c4367ca8"
I20260928 15:07:36.721311 29303 raft_consensus.cc:3037] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:36.721311 29434 raft_consensus.cc:3037] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:36.724619 29434 raft_consensus.cc:2454] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5b4cc19535491fa3c2beb9bc4e8000 in term 1.
I20260928 15:07:36.724622 29303 raft_consensus.cc:2454] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5b4cc19535491fa3c2beb9bc4e8000 in term 1.
I20260928 15:07:36.725061 29498 leader_election.cc:302] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 500ea281f2ce4034960472c2bb1b2892, 9b5b4cc19535491fa3c2beb9bc4e8000; no voters:
I20260928 15:07:36.725458 29635 raft_consensus.cc:2781] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:36.725953 29635 raft_consensus.cc:686] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 1 LEADER]: Becoming Leader. State: Replica: 9b5b4cc19535491fa3c2beb9bc4e8000, State: Running, Role: LEADER
I20260928 15:07:36.726487 29635 consensus_queue.cc:237] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } }
I20260928 15:07:36.731001 29161 catalog_manager.cc:5495] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b5b4cc19535491fa3c2beb9bc4e8000 (127.9.12.195). New cstate: current_term: 1 leader_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5b4cc19535491fa3c2beb9bc4e8000" member_type: VOTER last_known_addr { host: "127.9.12.195" port: 44221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 } health_report { overall_health: UNKNOWN } } }
I20260928 15:07:36.770678 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.765230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.9.12.195:44221)
0928 15:07:36.765496 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.765501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.765518 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:07:36.765723 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.765727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.765804 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.765928 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:36.766238 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.766973 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.768371 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.768819 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.768927 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.769437 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:36.769997 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:36.770119 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.770274 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.770348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 15:07:36.779279 29434 raft_consensus.cc:1261] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 1 FOLLOWER]: Refusing update from remote peer 9b5b4cc19535491fa3c2beb9bc4e8000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:36.779282 29303 raft_consensus.cc:1261] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 1 FOLLOWER]: Refusing update from remote peer 9b5b4cc19535491fa3c2beb9bc4e8000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:36.780333 29635 consensus_queue.cc:1035] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485a651d3afb473ebc747e44c4367ca8" member_type: VOTER last_known_addr { host: "127.9.12.193" port: 38243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:36.780637 29637 consensus_queue.cc:1035] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500ea281f2ce4034960472c2bb1b2892" member_type: VOTER last_known_addr { host: "127.9.12.194" port: 46351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:36.792105 29640 mvcc.cc:205] Tried to move back new op lower bound from 7334330600556699648 to 7334330600355807232. Current Snapshot: MvccSnapshot[applied={T|T < 7334330600556699648}]
I20260928 15:07:36.793402 29642 mvcc.cc:205] Tried to move back new op lower bound from 7334330600556699648 to 7334330600355807232. Current Snapshot: MvccSnapshot[applied={T|T < 7334330600556699648}]
I20260928 15:07:36.811789 29161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:36.812069 29161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:36.815585 29161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 050660a30a784dbea500f8f3440c4425: Sending DeleteTablet for 3 replicas of tablet 7753fc01ca8c470eadd2adc720d63f5e
I20260928 15:07:36.816674 29544 tablet_service.cc:1514] Processing DeleteTablet for tablet 7753fc01ca8c470eadd2adc720d63f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:36 UTC) from {username='slave'} at 127.0.0.1:33202
I20260928 15:07:36.816829 29414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7753fc01ca8c470eadd2adc720d63f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:36 UTC) from {username='slave'} at 127.0.0.1:42248
I20260928 15:07:36.816943 29283 tablet_service.cc:1514] Processing DeleteTablet for tablet 7753fc01ca8c470eadd2adc720d63f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:36 UTC) from {username='slave'} at 127.0.0.1:39154
I20260928 15:07:36.824286 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29221
I20260928 15:07:36.827816 29650 tablet_replica.cc:330] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: stopping tablet replica
I20260928 15:07:36.828563 29650 raft_consensus.cc:2229] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:36.829515 29650 raft_consensus.cc:2258] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:36.829916 29651 tablet_replica.cc:330] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: stopping tablet replica
I20260928 15:07:36.830323 29651 raft_consensus.cc:2229] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:36.830847 29651 raft_consensus.cc:2258] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:36.832010 29650 ts_tablet_manager.cc:1905] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:36.833048 29651 ts_tablet_manager.cc:1905] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:36.833658 29652 tablet_replica.cc:330] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: stopping tablet replica
I20260928 15:07:36.840077 29651 ts_tablet_manager.cc:1918] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:36.840250 29651 log.cc:1196] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-1/wal/wals/7753fc01ca8c470eadd2adc720d63f5e
I20260928 15:07:36.840744 29652 raft_consensus.cc:2229] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:36.840823 29651 ts_tablet_manager.cc:1939] T 7753fc01ca8c470eadd2adc720d63f5e P 500ea281f2ce4034960472c2bb1b2892: Deleting consensus metadata
I20260928 15:07:36.841356 29652 raft_consensus.cc:2258] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:36.840103 29650 ts_tablet_manager.cc:1918] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:36.842034 29650 log.cc:1196] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Deleting WAL directory at /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607741729566-9267-0/minicluster-data/ts-2/wal/wals/7753fc01ca8c470eadd2adc720d63f5e
I20260928 15:07:36.843556 29652 ts_tablet_manager.cc:1905] T 7753fc01ca8c470eadd2adc720d63f5e P 485a651d3afb473ebc747e44c4367ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:36.843518 29147 catalog_manager.cc:4841] TS 500ea281f2ce4034960472c2bb1b2892 (127.9.12.194:46351): tablet 7753fc01ca8c470eadd2adc720d63f5e (table test-table [id=e0eaeb884100433a9b8a88f08cc4ecf0]) successfully deleted
I20260928 15:07:36.844480 29650 ts_tablet_manager.cc:1939] T 7753fc01ca8c470eadd2adc720d63f5e P 9b5b4cc19535491fa3c2beb9bc4e8000: Deleting consensus metadata
I20260928 15:07:36.846257 29146 catalog_manager.cc:4841] TS 9b5b4cc19535491fa3c2beb9bc4e8000 (127.9.12.195:44221): tablet 7753fc01ca8c470eadd2adc720d63f5e (table test-table [id=e0eaeb884100433a9b8a88f08cc4ecf0]) successfully deleted
I20260928 15:07:36.847040 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.825426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.9.12.193:38243)
0928 15:07:36.825630 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.825638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.837967 (+ 12329us) server_negotiation.cc:407] Connection header received
0928 15:07:36.838468 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.838474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.838603 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.838684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.839380 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.840604 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.842613 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.843335 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.843449 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.845035 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.845047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.845064 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.845893 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.845961 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.845993 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.846087 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.846519 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.846633 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:07:36.847826 29343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:36.954346 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29221
I20260928 15:07:36.971529 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29352
W20260928 15:07:36.971588 29149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.12.193:38243 (error 108)
I20260928 15:07:36.973637 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.973148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39168)
0928 15:07:36.973317 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:36.973463 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:07:36.973908 29149 catalog_manager.cc:4584] TS 485a651d3afb473ebc747e44c4367ca8 (127.9.12.193:38243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7753fc01ca8c470eadd2adc720d63f5e: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
I20260928 15:07:36.980152 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.972433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.9.12.194:46351)
0928 15:07:36.972660 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.972666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.972910 (+ 244us) server_negotiation.cc:407] Connection header received
0928 15:07:36.973192 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.973198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.973323 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.973436 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.974012 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.974842 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.976782 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.977200 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.977295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.978532 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.978541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.978555 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.979313 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.979360 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.979381 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.979476 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.979657 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.979754 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 15:07:36.980903 29474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:37.027618 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.027044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39174)
0928 15:07:37.027284 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.027410 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 15:07:37.028801 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.028413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39188)
0928 15:07:37.028578 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.028663 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:07:37.077418 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29352
I20260928 15:07:37.089188 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.088741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39190)
0928 15:07:37.088948 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.089031 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:07:37.090243 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.089904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39206)
0928 15:07:37.090059 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.090140 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:07:37.100870 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29482
I20260928 15:07:37.113544 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.101709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.9.12.195:44221)
0928 15:07:37.101879 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:37.101885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:37.101903 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:07:37.102098 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:37.102102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:37.102210 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:37.102297 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:37.102771 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:37.103585 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:37.105426 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:37.105951 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:37.106142 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:37.111990 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:37.112000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:37.112014 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:37.112716 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:37.112766 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:37.112789 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:37.112873 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:37.113062 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 15:07:37.113151 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:07:37.114332 29604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:37.171203 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.170670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39214)
0928 15:07:37.170862 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.171024 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 15:07:37.172551 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.172112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39226)
0928 15:07:37.172346 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.172442 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 15:07:37.239509 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29482
I20260928 15:07:37.258837 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29131
I20260928 15:07:37.260007 29192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:37.372820 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.371996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.193:38243 (local address 127.0.0.1:39230)
0928 15:07:37.372404 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:37.372546 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.193:38243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260928 15:07:37.372875 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29131
2026-09-28T15:07:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2821 ms)
[----------] 6 tests from EncryptionPolicyTest (22211 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:37 dist-test-slave-dmhs krb5kdc[29661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:37 dist-test-slave-dmhs krb5kdc[29661](info): set up 2 sockets
Sep 28 15:07:37 dist-test-slave-dmhs krb5kdc[29661](info): commencing operation
krb5kdc: starting...
W20260928 15:07:38.481143 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:38 dist-test-slave-dmhs krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:38.752180 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:39.090453 29674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:39.090732 29674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:39.090833 29674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:39.097833 29674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:39.097945 29674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:39.097990 29674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:39.098042 29674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:39.098076 29674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:39.107568 29674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:39.108799 29674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:39.110087 29674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:39.121791 29680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:39.121804 29679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:39.122308 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:39.124020 29674 server_base.cc:1034] running on GCE node
I20260928 15:07:39.124858 29674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:39.126302 29674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:39.127537 29674 hybrid_clock.cc:648] HybridClock initialized: now 1790608059127486 us; error 67 us; skew 500 ppm
I20260928 15:07:39.127890 29674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:39 dist-test-slave-dmhs krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:39.131160 29674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:39.132810 29674 webserver.cc:460] Webserver started at http://127.0.0.1:37425/ using document root <none> and password file <none>
I20260928 15:07:39.133555 29674 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:39.133704 29674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:39.134022 29674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:39.137174 29674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "52263cff147d400cb60e2ec644cca2b3"
format_stamp: "Formatted at 2026-09-28 15:07:39 on dist-test-slave-dmhs"
server_key: "38bede21824ba9c25bc2f878868a3abc"
server_key_iv: "6368c8d0431a3253e619aeb5f8bd5c91"
server_key_version: "encryptionkey@0"
I20260928 15:07:39.137984 29674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "52263cff147d400cb60e2ec644cca2b3"
format_stamp: "Formatted at 2026-09-28 15:07:39 on dist-test-slave-dmhs"
server_key: "38bede21824ba9c25bc2f878868a3abc"
server_key_iv: "6368c8d0431a3253e619aeb5f8bd5c91"
server_key_version: "encryptionkey@0"
I20260928 15:07:39.143215 29674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:07:39.146643 29689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:39.148046 29674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:07:39.148227 29674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "52263cff147d400cb60e2ec644cca2b3"
format_stamp: "Formatted at 2026-09-28 15:07:39 on dist-test-slave-dmhs"
server_key: "38bede21824ba9c25bc2f878868a3abc"
server_key_iv: "6368c8d0431a3253e619aeb5f8bd5c91"
server_key_version: "encryptionkey@0"
I20260928 15:07:39.148478 29674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:39.169587 29674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:39.174041 29674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:39.174351 29674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:39.192941 29674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40337
I20260928 15:07:39.192976 29740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40337 every 8 connection(s)
I20260928 15:07:39.194339 29674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:39.196758 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29674
I20260928 15:07:39.197001 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:39.197827 9267 external_mini_cluster.cc:1428] Setting key 1294f40ba86183e871e8d252aca01096
I20260928 15:07:39.198740 29741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:07:39 dist-test-slave-dmhs krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:39.213994 29741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3: Bootstrap starting.
I20260928 15:07:39.217612 29741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:39.218918 29741 log.cc:824] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:39.221835 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.199794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40627 (local address 127.0.0.1:40337)
0928 15:07:39.200621 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.200632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.200673 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:07:39.202227 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.202247 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.202644 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.202976 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.204816 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.206174 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.207981 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.208733 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.208874 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.214077 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.214091 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.214130 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.214192 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.217318 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.218352 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.218359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.218365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.218538 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.219096 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.219101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.219107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.219818 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.220163 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.220516 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.220816 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":357,"threads_started":1}
I20260928 15:07:39.222074 29741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3: No bootstrap required, opened a new log
I20260928 15:07:39.225735 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:39.232060 29741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52263cff147d400cb60e2ec644cca2b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260928 15:07:39.232656 29741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:39.232744 29741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52263cff147d400cb60e2ec644cca2b3, State: Initialized, Role: FOLLOWER
I20260928 15:07:39.233372 29741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52263cff147d400cb60e2ec644cca2b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260928 15:07:39.233611 29741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:39.233741 29741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:39.233913 29741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:39.236920 29741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52263cff147d400cb60e2ec644cca2b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260928 15:07:39.237576 29741 leader_election.cc:302] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52263cff147d400cb60e2ec644cca2b3; no voters:
I20260928 15:07:39.238268 29741 leader_election.cc:288] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:39.238397 29751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:39.240541 29741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:39.243413 29751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [term 1 LEADER]: Becoming Leader. State: Replica: 52263cff147d400cb60e2ec644cca2b3, State: Running, Role: LEADER
I20260928 15:07:39.244300 29751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52263cff147d400cb60e2ec644cca2b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260928 15:07:39.247849 29753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52263cff147d400cb60e2ec644cca2b3. Latest consensus state: current_term: 1 leader_uuid: "52263cff147d400cb60e2ec644cca2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52263cff147d400cb60e2ec644cca2b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } } }
I20260928 15:07:39.248852 29753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:39.248456 29752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52263cff147d400cb60e2ec644cca2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52263cff147d400cb60e2ec644cca2b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } } }
I20260928 15:07:39.249531 29752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:39.250631 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.231545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51423 (local address 127.0.0.1:40337)
0928 15:07:39.231738 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.231744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.232136 (+ 392us) server_negotiation.cc:407] Connection header received
0928 15:07:39.232294 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.232299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.232376 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.232489 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.234155 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.235398 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.236709 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.237398 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.237531 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.241672 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.241684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.241692 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.241745 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.246063 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.247232 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.247237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.247240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.247410 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.248065 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.248083 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.248086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.248512 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.248861 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.249789 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.250039 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":65}
I20260928 15:07:39.257037 29764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:39.261874 29764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:39.271023 29764 catalog_manager.cc:1347] Generated new cluster ID: b5e1f3cd42034e2d9ac2e2d39db9d59b
I20260928 15:07:39.271142 29764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:39.287981 29764 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:39.289356 29764 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:39.298494 29764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52263cff147d400cb60e2ec644cca2b3: Generated new TSK 0
I20260928 15:07:39.299286 29764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:39.313207 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29674
I20260928 15:07:39.315924 29736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:39.407579 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29674
W20260928 15:07:39.425079 29659 connection.cc:535] client connection to 127.0.0.1:40337 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40337): Connection reset by peer (error 104)
2026-09-28T15:07:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:39 dist-test-slave-dmhs krb5kdc[29774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:39 dist-test-slave-dmhs krb5kdc[29774](info): set up 2 sockets
Sep 28 15:07:39 dist-test-slave-dmhs krb5kdc[29774](info): commencing operation
krb5kdc: starting...
W20260928 15:07:40.488910 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:40 dist-test-slave-dmhs krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:40.741865 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:41.066669 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:41.067030 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:41.067189 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:41.074458 29787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:41.074636 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:41.074743 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:41.074859 29787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:41.074942 29787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:41.084693 29787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:41.086009 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:41.087389 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:41.098944 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:41.098898 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:41.098990 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:41.101554 29787 server_base.cc:1034] running on GCE node
I20260928 15:07:41.102296 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:41.103549 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:41.104760 29787 hybrid_clock.cc:648] HybridClock initialized: now 1790608061104729 us; error 45 us; skew 500 ppm
I20260928 15:07:41.105090 29787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:41 dist-test-slave-dmhs krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:41.107939 29787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:41.109453 29787 webserver.cc:460] Webserver started at http://127.0.0.1:42313/ using document root <none> and password file <none>
I20260928 15:07:41.110140 29787 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:41.110285 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:41.110601 29787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:41.113457 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "6bfca20683af4ebb939ae0a737054f80"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-dmhs"
server_key: "5f41b0651db9747344a0585453ddce36"
server_key_iv: "f2c9f220645d018be7cfb8c00bf03626"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.114187 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "6bfca20683af4ebb939ae0a737054f80"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-dmhs"
server_key: "5f41b0651db9747344a0585453ddce36"
server_key_iv: "f2c9f220645d018be7cfb8c00bf03626"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.119653 29787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:07:41.122900 29802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:41.124075 29787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:07:41.124240 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "6bfca20683af4ebb939ae0a737054f80"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-dmhs"
server_key: "5f41b0651db9747344a0585453ddce36"
server_key_iv: "f2c9f220645d018be7cfb8c00bf03626"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.124485 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:41.137169 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:41.140913 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:41.141182 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:41.159212 29787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33311
I20260928 15:07:41.159247 29853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33311 every 8 connection(s)
I20260928 15:07:41.160610 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:41.164106 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:41.168023 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29787
I20260928 15:07:41.168197 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:41.168784 9267 external_mini_cluster.cc:1428] Setting key 756b9a4f37935e596e8a727e79f7e41c
I20260928 15:07:41.176792 29854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80: Bootstrap starting.
I20260928 15:07:41.180395 29854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:41.181592 29854 log.cc:824] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:41.181875 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.170539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36517 (local address 127.0.0.1:33311)
0928 15:07:41.171310 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:41.171328 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:41.171368 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:07:41.173014 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:41.173036 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:41.173466 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:41.173774 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:41.174359 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.175646 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.177416 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.178040 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.178189 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.179603 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:41.179610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:41.179622 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:41.180122 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:41.180193 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:41.180221 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:41.180483 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:41.180744 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 15:07:41.180925 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":378,"threads_started":1}
I20260928 15:07:41.184653 29854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80: No bootstrap required, opened a new log
I20260928 15:07:41.185636 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:41.193259 29854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfca20683af4ebb939ae0a737054f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260928 15:07:41.193704 29854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:41.193811 29854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bfca20683af4ebb939ae0a737054f80, State: Initialized, Role: FOLLOWER
I20260928 15:07:41.194445 29854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfca20683af4ebb939ae0a737054f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260928 15:07:41.194757 29854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:41.194906 29854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:41.195103 29854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:41.196621 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.190023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35187 (local address 127.0.0.1:33311)
0928 15:07:41.190250 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:41.190258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:41.190537 (+ 279us) server_negotiation.cc:407] Connection header received
0928 15:07:41.190706 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:41.190710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:41.190763 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:41.190831 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:41.191387 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.192152 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.194013 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.194448 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.194535 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.195791 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:41.195795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:41.195798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:41.195816 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:41.195869 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:41.195883 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:41.195972 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:41.196180 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 15:07:41.196222 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:07:41.197881 29854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfca20683af4ebb939ae0a737054f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260928 15:07:41.198361 29854 leader_election.cc:302] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bfca20683af4ebb939ae0a737054f80; no voters:
I20260928 15:07:41.198884 29854 leader_election.cc:288] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:41.199031 29864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:41.200450 29864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [term 1 LEADER]: Becoming Leader. State: Replica: 6bfca20683af4ebb939ae0a737054f80, State: Running, Role: LEADER
I20260928 15:07:41.200950 29864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfca20683af4ebb939ae0a737054f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260928 15:07:41.201140 29854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:41.203570 29866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bfca20683af4ebb939ae0a737054f80. Latest consensus state: current_term: 1 leader_uuid: "6bfca20683af4ebb939ae0a737054f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfca20683af4ebb939ae0a737054f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } } }
I20260928 15:07:41.203594 29865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bfca20683af4ebb939ae0a737054f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfca20683af4ebb939ae0a737054f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } } }
I20260928 15:07:41.203999 29866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:41.203999 29865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:41.207882 29870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:41.213021 29870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:41.221836 29870 catalog_manager.cc:1347] Generated new cluster ID: 944b0a49a9a14d7d9158afac0c7878be
I20260928 15:07:41.221956 29870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:41.239346 29870 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:41.240970 29870 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:41.247973 29870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bfca20683af4ebb939ae0a737054f80: Generated new TSK 0
I20260928 15:07:41.248878 29870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:41.255154 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29787
W20260928 15:07:41.256202 29849 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36517
W20260928 15:07:41.256953 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:41.257156 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29787
W20260928 15:07:41.273910 29772 connection.cc:535] client connection to 127.0.0.1:33311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33311): Connection reset by peer (error 104)
2026-09-28T15:07:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:41 dist-test-slave-dmhs krb5kdc[29885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:41 dist-test-slave-dmhs krb5kdc[29885](info): set up 2 sockets
Sep 28 15:07:41 dist-test-slave-dmhs krb5kdc[29885](info): commencing operation
krb5kdc: starting...
W20260928 15:07:42.360967 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:42 dist-test-slave-dmhs krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:42.607868 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:42.930347 29898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:42.930616 29898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:42.930802 29898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:42.937314 29898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:42.937438 29898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:42.937544 29898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:42.937646 29898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:42.937739 29898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:42.946866 29898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:42.948122 29898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:42.949424 29898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:42.960868 29903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:42.960932 29906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:42.960927 29904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:42.961848 29898 server_base.cc:1034] running on GCE node
I20260928 15:07:42.962564 29898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:42.963882 29898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:42.965107 29898 hybrid_clock.cc:648] HybridClock initialized: now 1790608062965059 us; error 60 us; skew 500 ppm
I20260928 15:07:42.965430 29898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:42 dist-test-slave-dmhs krb5kdc[29885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:42.968494 29898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:42.969898 29898 webserver.cc:460] Webserver started at http://127.0.0.1:37699/ using document root <none> and password file <none>
I20260928 15:07:42.970665 29898 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:42.970808 29898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:42.971110 29898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:42.973938 29898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "4b1bbbf7467342dfb32b3767061c21d6"
format_stamp: "Formatted at 2026-09-28 15:07:42 on dist-test-slave-dmhs"
server_key: "b3a898a128eb143ff61e3bb0d7184082"
server_key_iv: "252d37a1b861554482c9270f044f81f1"
server_key_version: "encryptionkey@0"
I20260928 15:07:42.974650 29898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "4b1bbbf7467342dfb32b3767061c21d6"
format_stamp: "Formatted at 2026-09-28 15:07:42 on dist-test-slave-dmhs"
server_key: "b3a898a128eb143ff61e3bb0d7184082"
server_key_iv: "252d37a1b861554482c9270f044f81f1"
server_key_version: "encryptionkey@0"
I20260928 15:07:42.979468 29898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 15:07:42.982828 29913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:42.984036 29898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:42.984213 29898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "4b1bbbf7467342dfb32b3767061c21d6"
format_stamp: "Formatted at 2026-09-28 15:07:42 on dist-test-slave-dmhs"
server_key: "b3a898a128eb143ff61e3bb0d7184082"
server_key_iv: "252d37a1b861554482c9270f044f81f1"
server_key_version: "encryptionkey@0"
I20260928 15:07:42.984465 29898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:42.989905 29898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:42.993616 29898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:42.993896 29898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:43.011622 29898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41705
I20260928 15:07:43.011667 29964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41705 every 8 connection(s)
I20260928 15:07:43.013090 29898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:43.016741 29965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:43.021589 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 29898
I20260928 15:07:43.021756 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:43.022212 9267 external_mini_cluster.cc:1428] Setting key 9982b28b02c13e15dc34119afd326aa8
I20260928 15:07:43.028993 29965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6: Bootstrap starting.
I20260928 15:07:43.031926 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.024072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51885 (local address 127.0.0.1:41705)
0928 15:07:43.024915 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.024929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.024987 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:07:43.027782 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.027831 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.028306 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.028759 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:43.029865 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:43.029874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:43.029892 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:43.029911 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:43.029990 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:43.030019 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:43.030244 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.030387 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.030599 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":366,"threads_started":1}
I20260928 15:07:43.033085 29965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:43.034245 29965 log.cc:824] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:43.035503 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:43.036931 29965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6: No bootstrap required, opened a new log
I20260928 15:07:43.042688 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.040022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44105 (local address 127.0.0.1:41705)
0928 15:07:43.040191 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.040196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.040752 (+ 556us) server_negotiation.cc:407] Connection header received
0928 15:07:43.040883 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.040888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.040970 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.041072 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:43.042012 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:43.042018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:43.042023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:43.042040 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:43.042095 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:43.042115 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:43.042225 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.042326 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.042384 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:07:43.044991 29965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1bbbf7467342dfb32b3767061c21d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260928 15:07:43.045346 29965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:43.045449 29965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1bbbf7467342dfb32b3767061c21d6, State: Initialized, Role: FOLLOWER
I20260928 15:07:43.045984 29965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1bbbf7467342dfb32b3767061c21d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260928 15:07:43.046216 29965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:43.046342 29965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:43.046509 29965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:43.048890 29965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1bbbf7467342dfb32b3767061c21d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260928 15:07:43.049389 29965 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1bbbf7467342dfb32b3767061c21d6; no voters:
I20260928 15:07:43.050051 29965 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:43.050211 29975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:43.050972 29975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [term 1 LEADER]: Becoming Leader. State: Replica: 4b1bbbf7467342dfb32b3767061c21d6, State: Running, Role: LEADER
I20260928 15:07:43.051488 29975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1bbbf7467342dfb32b3767061c21d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260928 15:07:43.053347 29975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1bbbf7467342dfb32b3767061c21d6. Latest consensus state: current_term: 1 leader_uuid: "4b1bbbf7467342dfb32b3767061c21d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1bbbf7467342dfb32b3767061c21d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260928 15:07:43.053615 29975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:43.054261 29977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b1bbbf7467342dfb32b3767061c21d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1bbbf7467342dfb32b3767061c21d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260928 15:07:43.054724 29977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:43.056869 29965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:43.057087 29979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:43.063009 29979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:43.072003 29979 catalog_manager.cc:1347] Generated new cluster ID: 4937e578aa1542f29ea57413d07d83c1
I20260928 15:07:43.072136 29979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:43.082847 29979 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:43.083256 29979 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:43.093712 29979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b1bbbf7467342dfb32b3767061c21d6: Generated new TSK 0
I20260928 15:07:43.094375 29979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:43.105897 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29898
W20260928 15:07:43.107062 29960 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51885
W20260928 15:07:43.107710 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 29898: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:43.107834 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 29898
2026-09-28T15:07:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:43 dist-test-slave-dmhs krb5kdc[29996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:43 dist-test-slave-dmhs krb5kdc[29996](info): set up 2 sockets
Sep 28 15:07:43 dist-test-slave-dmhs krb5kdc[29996](info): commencing operation
krb5kdc: starting...
W20260928 15:07:44.188832 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:44 dist-test-slave-dmhs krb5kdc[29996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:44.434696 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:44.759907 30009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:44.760277 30009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:44.760453 30009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:44.767755 30009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:44.767915 30009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:44.768021 30009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:44.768146 30009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:44.768220 30009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:44.778152 30009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:44.779467 30009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:44.780789 30009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:44.792158 30015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:44.792218 30017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:44.792241 30014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:44.793023 30009 server_base.cc:1034] running on GCE node
I20260928 15:07:44.793713 30009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:44.795055 30009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:44.796339 30009 hybrid_clock.cc:648] HybridClock initialized: now 1790608064796240 us; error 46 us; skew 500 ppm
I20260928 15:07:44.796669 30009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:44 dist-test-slave-dmhs krb5kdc[29996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:44.799682 30009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:44.801160 30009 webserver.cc:460] Webserver started at http://127.0.0.1:34671/ using document root <none> and password file <none>
I20260928 15:07:44.801791 30009 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:44.801927 30009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:44.802207 30009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:44.805260 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "ac3c02600d6d4db1b5efd384cf475134"
format_stamp: "Formatted at 2026-09-28 15:07:44 on dist-test-slave-dmhs"
server_key: "d8c0f3c8934796f4294fdf8313b45ee3"
server_key_iv: "958381b3066620803f670848d36f4d3e"
server_key_version: "encryptionkey@0"
I20260928 15:07:44.806032 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "ac3c02600d6d4db1b5efd384cf475134"
format_stamp: "Formatted at 2026-09-28 15:07:44 on dist-test-slave-dmhs"
server_key: "d8c0f3c8934796f4294fdf8313b45ee3"
server_key_iv: "958381b3066620803f670848d36f4d3e"
server_key_version: "encryptionkey@0"
I20260928 15:07:44.810885 30009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 15:07:44.813948 30024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:44.815071 30009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:07:44.815277 30009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "ac3c02600d6d4db1b5efd384cf475134"
format_stamp: "Formatted at 2026-09-28 15:07:44 on dist-test-slave-dmhs"
server_key: "d8c0f3c8934796f4294fdf8313b45ee3"
server_key_iv: "958381b3066620803f670848d36f4d3e"
server_key_version: "encryptionkey@0"
I20260928 15:07:44.815461 30009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:44.833851 30009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:44.837934 30009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:44.838213 30009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:44.856387 30009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41693
I20260928 15:07:44.856390 30075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41693 every 8 connection(s)
I20260928 15:07:44.857795 30009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:44.861392 30076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:44.867702 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30009
I20260928 15:07:44.867923 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:44.868515 9267 external_mini_cluster.cc:1428] Setting key f2ead9e2b96dbcde0365f5a9399e74c9
I20260928 15:07:44.871975 30076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134: Bootstrap starting.
I20260928 15:07:44.875501 30076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:44.876598 30076 log.cc:824] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:44.879488 30076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134: No bootstrap required, opened a new log
Sep 28 15:07:44 dist-test-slave-dmhs krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:44.887687 30076 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3c02600d6d4db1b5efd384cf475134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260928 15:07:44.888069 30076 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:44.888216 30076 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3c02600d6d4db1b5efd384cf475134, State: Initialized, Role: FOLLOWER
I20260928 15:07:44.888913 30076 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3c02600d6d4db1b5efd384cf475134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260928 15:07:44.889191 30076 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:44.889348 30076 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:44.889561 30076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:44.891042 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:44.870265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46911 (local address 127.0.0.1:41693)
0928 15:07:44.871021 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:44.871033 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:44.871071 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:07:44.872781 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:44.872802 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:44.873165 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:44.873469 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:44.875110 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:44.876418 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:44.877960 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:44.878601 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:44.878725 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:44.883604 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:44.883611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:44.883624 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:44.883657 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:44.886671 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:44.887488 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:44.887495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:44.887501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:44.887658 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:44.888091 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:44.888094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:44.888096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:44.888814 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:44.889149 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:44.889589 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 15:07:44.890024 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":371,"threads_started":1}
I20260928 15:07:44.892390 30076 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3c02600d6d4db1b5efd384cf475134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260928 15:07:44.892910 30076 leader_election.cc:302] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3c02600d6d4db1b5efd384cf475134; no voters:
I20260928 15:07:44.893541 30076 leader_election.cc:288] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:44.893648 30081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:44.895107 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:44.895213 30081 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [term 1 LEADER]: Becoming Leader. State: Replica: ac3c02600d6d4db1b5efd384cf475134, State: Running, Role: LEADER
I20260928 15:07:44.895731 30076 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:44.895738 30081 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3c02600d6d4db1b5efd384cf475134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260928 15:07:44.900179 30082 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac3c02600d6d4db1b5efd384cf475134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3c02600d6d4db1b5efd384cf475134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } } }
I20260928 15:07:44.900578 30082 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:44.901089 30083 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3c02600d6d4db1b5efd384cf475134. Latest consensus state: current_term: 1 leader_uuid: "ac3c02600d6d4db1b5efd384cf475134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3c02600d6d4db1b5efd384cf475134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } } }
I20260928 15:07:44.901445 30083 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:44.905226 30089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:44.911549 30089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:44.932384 30089 catalog_manager.cc:1347] Generated new cluster ID: 39832cc0deea46dca8d149a21710fd49
I20260928 15:07:44.932508 30089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:44.951913 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:44.931253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45327 (local address 127.0.0.1:41693)
0928 15:07:44.931431 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:44.931437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:44.933116 (+ 1679us) server_negotiation.cc:407] Connection header received
0928 15:07:44.933352 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:44.933357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:44.933448 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:44.933562 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:44.937215 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:44.938672 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:44.941197 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:44.941934 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:44.942080 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:44.945061 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:44.945068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:44.945073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:44.945113 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:44.948911 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:44.949692 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:44.949698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:44.949701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:44.949866 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:44.950283 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:44.950289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:44.950292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:44.950709 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:44.950982 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:44.951291 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 15:07:44.951516 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260928 15:07:44.955606 30089 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:44.957299 30089 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:44.966661 30089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3c02600d6d4db1b5efd384cf475134: Generated new TSK 0
I20260928 15:07:44.967468 30089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:44.977404 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30009
I20260928 15:07:44.978471 30071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:45.069069 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30009
2026-09-28T15:07:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:45 dist-test-slave-dmhs krb5kdc[30109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:45 dist-test-slave-dmhs krb5kdc[30109](info): set up 2 sockets
Sep 28 15:07:45 dist-test-slave-dmhs krb5kdc[30109](info): commencing operation
krb5kdc: starting...
W20260928 15:07:46.144986 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:46 dist-test-slave-dmhs krb5kdc[30109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:46.391485 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:46.717506 30122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:46.717832 30122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:46.718007 30122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:46.725447 30122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:46.725636 30122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:46.725736 30122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:46.725852 30122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:46.725939 30122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:46.735778 30122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:46.737169 30122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:46.738499 30122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:46.750278 30127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:46.750319 30128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:46.750232 30130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:46.751479 30122 server_base.cc:1034] running on GCE node
I20260928 15:07:46.752221 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:46.753623 30122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:46.754812 30122 hybrid_clock.cc:648] HybridClock initialized: now 1790608066754751 us; error 51 us; skew 500 ppm
I20260928 15:07:46.755154 30122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:46 dist-test-slave-dmhs krb5kdc[30109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:46.758184 30122 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:46.759534 30122 webserver.cc:460] Webserver started at http://127.0.0.1:34283/ using document root <none> and password file <none>
I20260928 15:07:46.760175 30122 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:46.760325 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:46.760604 30122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:46.763500 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "09c0364c03b5417b9438b64c23d62223"
format_stamp: "Formatted at 2026-09-28 15:07:46 on dist-test-slave-dmhs"
server_key: "d1f5c403765f0973532c5b6a414d04f5"
server_key_iv: "05bcceafe3a1cf0158d2b57c1a0d1341"
server_key_version: "encryptionkey@0"
I20260928 15:07:46.764165 30122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "09c0364c03b5417b9438b64c23d62223"
format_stamp: "Formatted at 2026-09-28 15:07:46 on dist-test-slave-dmhs"
server_key: "d1f5c403765f0973532c5b6a414d04f5"
server_key_iv: "05bcceafe3a1cf0158d2b57c1a0d1341"
server_key_version: "encryptionkey@0"
I20260928 15:07:46.769243 30122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 15:07:46.772470 30137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:46.773757 30122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:07:46.773931 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "09c0364c03b5417b9438b64c23d62223"
format_stamp: "Formatted at 2026-09-28 15:07:46 on dist-test-slave-dmhs"
server_key: "d1f5c403765f0973532c5b6a414d04f5"
server_key_iv: "05bcceafe3a1cf0158d2b57c1a0d1341"
server_key_version: "encryptionkey@0"
I20260928 15:07:46.774127 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:46.804677 30122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:46.808852 30122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:46.809134 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:46.827090 30122 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260928 15:07:46.827135 30188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260928 15:07:46.828498 30122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:46.831965 30189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:46.834475 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30122
I20260928 15:07:46.834656 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:46.835218 9267 external_mini_cluster.cc:1428] Setting key fbdfee295c752359790671406b672edf
I20260928 15:07:46.844089 30189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223: Bootstrap starting.
I20260928 15:07:46.847609 30189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:46.848158 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:46.836948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50305 (local address 127.0.0.1:42683)
0928 15:07:46.837772 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:46.837783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:46.837822 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:07:46.839358 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:46.839378 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:46.839775 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:46.840075 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:46.840721 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:46.841993 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:46.843837 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:46.844603 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:46.844733 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:46.845906 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:46.845914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:46.845931 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:46.846394 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:46.846496 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:46.846519 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:46.846599 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:46.846869 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:07:46.847128 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":420,"threads_started":1}
I20260928 15:07:46.848831 30189 log.cc:824] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:46.852146 30189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223: No bootstrap required, opened a new log
I20260928 15:07:46.852607 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:46.862414 30189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c0364c03b5417b9438b64c23d62223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260928 15:07:46.862897 30189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:46.863029 30189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c0364c03b5417b9438b64c23d62223, State: Initialized, Role: FOLLOWER
I20260928 15:07:46.863655 30189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c0364c03b5417b9438b64c23d62223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260928 15:07:46.863987 30189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:46.864151 30189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:46.864432 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:46.857942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43241 (local address 127.0.0.1:42683)
0928 15:07:46.858135 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:46.858141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:46.858518 (+ 377us) server_negotiation.cc:407] Connection header received
0928 15:07:46.858685 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:46.858691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:46.858782 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:46.858911 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:46.859351 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:46.860510 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:46.861776 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:46.862395 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:46.862529 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:46.863604 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:46.863612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:46.863619 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:46.863645 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:46.863690 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:46.863711 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:46.863802 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:46.863931 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:07:46.864024 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:07:46.864392 30189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:46.866843 30189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c0364c03b5417b9438b64c23d62223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260928 15:07:46.867455 30189 leader_election.cc:302] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c0364c03b5417b9438b64c23d62223; no voters:
I20260928 15:07:46.868090 30189 leader_election.cc:288] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:46.868193 30199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:46.868968 30199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [term 1 LEADER]: Becoming Leader. State: Replica: 09c0364c03b5417b9438b64c23d62223, State: Running, Role: LEADER
I20260928 15:07:46.869514 30199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c0364c03b5417b9438b64c23d62223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260928 15:07:46.870328 30189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:46.872501 30200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09c0364c03b5417b9438b64c23d62223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c0364c03b5417b9438b64c23d62223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260928 15:07:46.872834 30200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:46.873184 30201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c0364c03b5417b9438b64c23d62223. Latest consensus state: current_term: 1 leader_uuid: "09c0364c03b5417b9438b64c23d62223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c0364c03b5417b9438b64c23d62223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260928 15:07:46.873500 30201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:46.877290 30207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:46.882961 30207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:46.891803 30207 catalog_manager.cc:1347] Generated new cluster ID: 8b7681cbb02f4f67bb4f1bb13b3ec0e4
I20260928 15:07:46.891911 30207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:46.908682 30207 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:46.910568 30207 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:46.922437 30207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09c0364c03b5417b9438b64c23d62223: Generated new TSK 0
I20260928 15:07:46.923465 30207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:46.930264 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30122
W20260928 15:07:46.931411 30184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50305
W20260928 15:07:46.932090 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30122: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:46.932235 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30122
2026-09-28T15:07:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:47 dist-test-slave-dmhs krb5kdc[30220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:47 dist-test-slave-dmhs krb5kdc[30220](info): set up 2 sockets
Sep 28 15:07:47 dist-test-slave-dmhs krb5kdc[30220](info): commencing operation
krb5kdc: starting...
W20260928 15:07:48.016911 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:48 dist-test-slave-dmhs krb5kdc[30220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:48.257515 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:48.582356 30233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:48.582639 30233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:48.582729 30233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:48.589937 30233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:48.590054 30233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:48.590102 30233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:48.590159 30233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:48.590204 30233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:48.599980 30233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:48.601295 30233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:48.602583 30233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:48.614480 30238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:48.614456 30241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:48.614419 30239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:48.615228 30233 server_base.cc:1034] running on GCE node
I20260928 15:07:48.615993 30233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:48.617543 30233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:48.618777 30233 hybrid_clock.cc:648] HybridClock initialized: now 1790608068618690 us; error 78 us; skew 500 ppm
I20260928 15:07:48.619129 30233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:48 dist-test-slave-dmhs krb5kdc[30220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:48.631923 30233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:48.633589 30233 webserver.cc:460] Webserver started at http://127.0.0.1:41793/ using document root <none> and password file <none>
I20260928 15:07:48.634289 30233 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:48.634434 30233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:48.634743 30233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:48.637931 30233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "f222c8b8e0a14f8985480c5dc00f34d3"
format_stamp: "Formatted at 2026-09-28 15:07:48 on dist-test-slave-dmhs"
server_key: "7206877a5c7b8c83da5779de63eb2e4c"
server_key_iv: "e08560084a7240b95a4f6048bdb2cb93"
server_key_version: "encryptionkey@0"
I20260928 15:07:48.638705 30233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "f222c8b8e0a14f8985480c5dc00f34d3"
format_stamp: "Formatted at 2026-09-28 15:07:48 on dist-test-slave-dmhs"
server_key: "7206877a5c7b8c83da5779de63eb2e4c"
server_key_iv: "e08560084a7240b95a4f6048bdb2cb93"
server_key_version: "encryptionkey@0"
I20260928 15:07:48.643692 30233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:07:48.647003 30248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:48.648320 30233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:48.648494 30233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "f222c8b8e0a14f8985480c5dc00f34d3"
format_stamp: "Formatted at 2026-09-28 15:07:48 on dist-test-slave-dmhs"
server_key: "7206877a5c7b8c83da5779de63eb2e4c"
server_key_iv: "e08560084a7240b95a4f6048bdb2cb93"
server_key_version: "encryptionkey@0"
I20260928 15:07:48.648700 30233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:48.654222 30233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:48.658592 30233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:48.658872 30233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:48.677007 30233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37891
I20260928 15:07:48.677048 30299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37891 every 8 connection(s)
I20260928 15:07:48.678347 30233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:48.681918 30300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:48.688695 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30233
I20260928 15:07:48.688886 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:48.689487 9267 external_mini_cluster.cc:1428] Setting key 582cad507651a6a9f07d53f449c10466
I20260928 15:07:48.692492 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3: Bootstrap starting.
I20260928 15:07:48.695542 30300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:48.696743 30300 log.cc:824] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:48.697021 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:48.691192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43261 (local address 127.0.0.1:37891)
0928 15:07:48.691933 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:48.691945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:48.691982 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:07:48.693708 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:48.693733 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:48.694067 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:48.694427 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:48.695430 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:48.695436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:48.695447 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:48.695465 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:48.695518 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:48.695543 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:48.695734 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:48.695872 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 15:07:48.696053 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":342,"threads_started":1}
I20260928 15:07:48.699481 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3: No bootstrap required, opened a new log
I20260928 15:07:48.700569 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:48.707129 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:48.704634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51097 (local address 127.0.0.1:37891)
0928 15:07:48.704851 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:48.704859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:48.705299 (+ 440us) server_negotiation.cc:407] Connection header received
0928 15:07:48.705452 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:48.705455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:48.705495 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:48.705584 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:48.706477 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:48.706481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:48.706487 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:48.706500 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:48.706546 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:48.706560 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:48.706651 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:48.706772 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 15:07:48.706824 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:07:48.707940 30300 raft_consensus.cc:348] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260928 15:07:48.708389 30300 raft_consensus.cc:374] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:48.708494 30300 raft_consensus.cc:729] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f222c8b8e0a14f8985480c5dc00f34d3, State: Initialized, Role: FOLLOWER
I20260928 15:07:48.709051 30300 consensus_queue.cc:260] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260928 15:07:48.709295 30300 raft_consensus.cc:388] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:48.709445 30300 raft_consensus.cc:482] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:48.709632 30300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:48.711778 30300 raft_consensus.cc:504] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260928 15:07:48.712297 30300 leader_election.cc:302] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f222c8b8e0a14f8985480c5dc00f34d3; no voters:
I20260928 15:07:48.712913 30300 leader_election.cc:288] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:48.713039 30310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:48.713735 30310 raft_consensus.cc:686] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [term 1 LEADER]: Becoming Leader. State: Replica: f222c8b8e0a14f8985480c5dc00f34d3, State: Running, Role: LEADER
I20260928 15:07:48.714293 30310 consensus_queue.cc:237] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260928 15:07:48.715063 30300 sys_catalog.cc:564] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:48.717089 30311 sys_catalog.cc:455] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } } }
I20260928 15:07:48.717101 30312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f222c8b8e0a14f8985480c5dc00f34d3. Latest consensus state: current_term: 1 leader_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f222c8b8e0a14f8985480c5dc00f34d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } } }
I20260928 15:07:48.717540 30311 sys_catalog.cc:458] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:48.717540 30312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:48.719161 30316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:48.724547 30316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:48.733552 30316 catalog_manager.cc:1347] Generated new cluster ID: 24c0e529b0d24788904a43048e359037
I20260928 15:07:48.733703 30316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:48.756220 30316 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:48.756821 30316 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:48.765295 30316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f222c8b8e0a14f8985480c5dc00f34d3: Generated new TSK 0
I20260928 15:07:48.766196 30316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:48.771752 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30233
W20260928 15:07:48.772995 30295 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43261
W20260928 15:07:48.773710 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30233: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:48.773871 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30233
2026-09-28T15:07:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:48 dist-test-slave-dmhs krb5kdc[30331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:48 dist-test-slave-dmhs krb5kdc[30331](info): set up 2 sockets
Sep 28 15:07:48 dist-test-slave-dmhs krb5kdc[30331](info): commencing operation
krb5kdc: starting...
W20260928 15:07:49.856796 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:50 dist-test-slave-dmhs krb5kdc[30331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:50.111949 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:50.439464 30344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:50.439749 30344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:50.439843 30344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:50.448957 30344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:50.449116 30344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:50.449205 30344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:50.449298 30344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:50.449364 30344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:50.460512 30344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:50.461808 30344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:50.463106 30344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:50.474848 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:50.475003 30350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:50.474967 30352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:50.475711 30344 server_base.cc:1034] running on GCE node
I20260928 15:07:50.476554 30344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:50.477931 30344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:50.479130 30344 hybrid_clock.cc:648] HybridClock initialized: now 1790608070479068 us; error 51 us; skew 500 ppm
I20260928 15:07:50.479498 30344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:50 dist-test-slave-dmhs krb5kdc[30331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:50.482568 30344 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:50.483999 30344 webserver.cc:460] Webserver started at http://127.0.0.1:45349/ using document root <none> and password file <none>
I20260928 15:07:50.484748 30344 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:50.484892 30344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:50.485188 30344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:50.488129 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "2c94636449f64906bc0ce27f2e94c813"
format_stamp: "Formatted at 2026-09-28 15:07:50 on dist-test-slave-dmhs"
server_key: "8c940f1399e959bc58460c8f865d4f20"
server_key_iv: "6759e089df59f935cdc7f25cd10af179"
server_key_version: "encryptionkey@0"
I20260928 15:07:50.488952 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "2c94636449f64906bc0ce27f2e94c813"
format_stamp: "Formatted at 2026-09-28 15:07:50 on dist-test-slave-dmhs"
server_key: "8c940f1399e959bc58460c8f865d4f20"
server_key_iv: "6759e089df59f935cdc7f25cd10af179"
server_key_version: "encryptionkey@0"
I20260928 15:07:50.493932 30344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:07:50.497220 30359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:50.498498 30344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:07:50.498683 30344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "2c94636449f64906bc0ce27f2e94c813"
format_stamp: "Formatted at 2026-09-28 15:07:50 on dist-test-slave-dmhs"
server_key: "8c940f1399e959bc58460c8f865d4f20"
server_key_iv: "6759e089df59f935cdc7f25cd10af179"
server_key_version: "encryptionkey@0"
I20260928 15:07:50.498873 30344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:50.517035 30344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:50.521497 30344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:50.521800 30344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:50.540510 30344 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41663
I20260928 15:07:50.540534 30410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41663 every 8 connection(s)
I20260928 15:07:50.541913 30344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:50.545743 30411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:50.546921 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30344
I20260928 15:07:50.547119 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:50.547694 9267 external_mini_cluster.cc:1428] Setting key a6be2539b3c37396726c26a5ac77650a
I20260928 15:07:50.559785 30411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813: Bootstrap starting.
Sep 28 15:07:50 dist-test-slave-dmhs krb5kdc[30331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:50.563839 30411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:50.565348 30411 log.cc:824] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:50.568604 30411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813: No bootstrap required, opened a new log
I20260928 15:07:50.571638 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:50.549531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:49195 (local address 127.0.0.1:41663)
0928 15:07:50.550301 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:50.550313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:50.550364 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:07:50.551993 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:50.552018 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:50.552492 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:50.552813 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:50.554572 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:50.555819 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:50.557682 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:50.558335 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:50.558458 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:50.563251 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:50.563261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:50.563279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:50.563339 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:50.567071 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:50.567846 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:50.567854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:50.567862 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:50.568075 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:50.568536 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:50.568540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:50.568543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:50.569242 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:50.569586 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:50.569948 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 15:07:50.570279 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
I20260928 15:07:50.575385 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:50.577715 30411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94636449f64906bc0ce27f2e94c813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 15:07:50.578225 30411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:50.578361 30411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c94636449f64906bc0ce27f2e94c813, State: Initialized, Role: FOLLOWER
I20260928 15:07:50.579010 30411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94636449f64906bc0ce27f2e94c813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 15:07:50.579241 30411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:50.579392 30411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:50.579572 30411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:50.582335 30411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94636449f64906bc0ce27f2e94c813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 15:07:50.582942 30411 leader_election.cc:302] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c94636449f64906bc0ce27f2e94c813; no voters:
I20260928 15:07:50.583588 30411 leader_election.cc:288] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:50.583724 30421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:50.585669 30421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [term 1 LEADER]: Becoming Leader. State: Replica: 2c94636449f64906bc0ce27f2e94c813, State: Running, Role: LEADER
I20260928 15:07:50.585779 30411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:50.586261 30421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94636449f64906bc0ce27f2e94c813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 15:07:50.589136 30423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c94636449f64906bc0ce27f2e94c813. Latest consensus state: current_term: 1 leader_uuid: "2c94636449f64906bc0ce27f2e94c813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94636449f64906bc0ce27f2e94c813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } } }
I20260928 15:07:50.589530 30423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:50.590930 30422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c94636449f64906bc0ce27f2e94c813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94636449f64906bc0ce27f2e94c813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } } }
I20260928 15:07:50.591290 30422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:50.597673 30427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:50.598603 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:50.579762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:56723 (local address 127.0.0.1:41663)
0928 15:07:50.579956 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:50.579965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:50.580490 (+ 525us) server_negotiation.cc:407] Connection header received
0928 15:07:50.580610 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:50.580615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:50.580701 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:50.580800 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:50.582189 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:50.583284 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:50.585372 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:50.586100 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:50.586236 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:50.589577 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:50.589583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:50.589589 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:50.589629 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:50.594468 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:50.595744 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:50.595752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:50.595758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:50.595929 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:50.596578 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:50.596587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:50.596592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:50.597234 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:50.597609 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:50.597973 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:07:50.598147 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:07:50.604100 30427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:50.615590 30427 catalog_manager.cc:1347] Generated new cluster ID: 7f852d70acdb4fb6a73ba6c816d77dd6
I20260928 15:07:50.615710 30427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:50.657334 30427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:50.658690 30427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:50.670372 30427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c94636449f64906bc0ce27f2e94c813: Generated new TSK 0
I20260928 15:07:50.671209 30427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:50.679644 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30344
I20260928 15:07:50.681088 30406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:50.774012 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30344
W20260928 15:07:50.791320 30329 connection.cc:535] client connection to 127.0.0.1:41663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41663): Connection reset by peer (error 104)
2026-09-28T15:07:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:50 dist-test-slave-dmhs krb5kdc[30443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:50 dist-test-slave-dmhs krb5kdc[30443](info): set up 2 sockets
Sep 28 15:07:50 dist-test-slave-dmhs krb5kdc[30443](info): commencing operation
krb5kdc: starting...
W20260928 15:07:51.857003 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:52 dist-test-slave-dmhs krb5kdc[30443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:52.115432 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:52.465662 30456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:52.466013 30456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:52.466182 30456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:52.473474 30456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:52.473627 30456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:52.473729 30456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:52.473837 30456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:52.473920 30456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:52.484433 30456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:52.485810 30456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:52.487172 30456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:52.499392 30461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:52.499326 30464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:52.499352 30462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:52.500399 30456 server_base.cc:1034] running on GCE node
I20260928 15:07:52.501145 30456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:52.502547 30456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:52.503799 30456 hybrid_clock.cc:648] HybridClock initialized: now 1790608072503753 us; error 59 us; skew 500 ppm
I20260928 15:07:52.504187 30456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:52 dist-test-slave-dmhs krb5kdc[30443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:52.507321 30456 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:52.508960 30456 webserver.cc:460] Webserver started at http://127.0.0.1:33071/ using document root <none> and password file <none>
I20260928 15:07:52.509685 30456 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:52.509835 30456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:52.510138 30456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:52.513193 30456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "ba578420ffe14b9ba9b063afa3cb8ae5"
format_stamp: "Formatted at 2026-09-28 15:07:52 on dist-test-slave-dmhs"
server_key: "2b039f25ab022ba33d4456d26a0436c9"
server_key_iv: "2c6f96056ca79ee8336f856ee4f9e015"
server_key_version: "encryptionkey@0"
I20260928 15:07:52.513994 30456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "ba578420ffe14b9ba9b063afa3cb8ae5"
format_stamp: "Formatted at 2026-09-28 15:07:52 on dist-test-slave-dmhs"
server_key: "2b039f25ab022ba33d4456d26a0436c9"
server_key_iv: "2c6f96056ca79ee8336f856ee4f9e015"
server_key_version: "encryptionkey@0"
I20260928 15:07:52.519248 30456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:07:52.522603 30471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:52.523979 30456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:52.524185 30456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "ba578420ffe14b9ba9b063afa3cb8ae5"
format_stamp: "Formatted at 2026-09-28 15:07:52 on dist-test-slave-dmhs"
server_key: "2b039f25ab022ba33d4456d26a0436c9"
server_key_iv: "2c6f96056ca79ee8336f856ee4f9e015"
server_key_version: "encryptionkey@0"
I20260928 15:07:52.524475 30456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:52.538391 30456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:52.542430 30456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:52.542711 30456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:52.561425 30456 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39593
I20260928 15:07:52.561451 30522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39593 every 8 connection(s)
I20260928 15:07:52.562810 30456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:52.566525 30523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:52.571225 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30456
I20260928 15:07:52.571420 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:52.571977 9267 external_mini_cluster.cc:1428] Setting key 0129b50f81280189176e7cf8402e1ce3
I20260928 15:07:52.577859 30523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5: Bootstrap starting.
I20260928 15:07:52.581527 30523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:52.582870 30523 log.cc:824] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:52.585886 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:52.573831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:55513 (local address 127.0.0.1:39593)
0928 15:07:52.574624 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:52.574635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:52.574681 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:07:52.576501 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:52.576523 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:52.576904 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:52.577201 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:52.577808 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:52.579161 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:52.581090 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:52.581855 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:52.582019 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:52.583295 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:52.583302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:52.583316 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:52.584132 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:52.584232 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:52.584288 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:52.584429 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:52.584741 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 15:07:52.584958 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":366,"threads_started":1}
I20260928 15:07:52.586004 30523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5: No bootstrap required, opened a new log
I20260928 15:07:52.589691 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:52.595626 30523 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260928 15:07:52.596117 30523 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:52.596221 30523 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba578420ffe14b9ba9b063afa3cb8ae5, State: Initialized, Role: FOLLOWER
I20260928 15:07:52.596964 30523 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260928 15:07:52.597260 30523 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:52.597404 30523 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:52.597589 30523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:52.600344 30523 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260928 15:07:52.600939 30523 leader_election.cc:302] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba578420ffe14b9ba9b063afa3cb8ae5; no voters:
I20260928 15:07:52.601642 30523 leader_election.cc:288] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:52.601765 30533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:52.601972 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:52.594949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:37117 (local address 127.0.0.1:39593)
0928 15:07:52.595147 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:52.595153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:52.595553 (+ 400us) server_negotiation.cc:407] Connection header received
0928 15:07:52.595676 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:52.595681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:52.595754 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:52.595861 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:52.596603 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:52.597769 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:52.599051 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:52.599749 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:52.599868 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:52.600964 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:52.600968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:52.600972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:52.600988 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:52.601044 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:52.601060 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:52.601243 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:52.601436 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:07:52.601495 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:07:52.603149 30533 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [term 1 LEADER]: Becoming Leader. State: Replica: ba578420ffe14b9ba9b063afa3cb8ae5, State: Running, Role: LEADER
I20260928 15:07:52.603790 30533 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260928 15:07:52.604107 30523 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:52.607069 30534 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } } }
I20260928 15:07:52.607045 30535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba578420ffe14b9ba9b063afa3cb8ae5. Latest consensus state: current_term: 1 leader_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba578420ffe14b9ba9b063afa3cb8ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } } }
I20260928 15:07:52.607515 30534 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:52.607515 30535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:52.608590 30542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:52.615353 30542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:52.626091 30542 catalog_manager.cc:1347] Generated new cluster ID: c204f201228747809ab57529bdea97fb
I20260928 15:07:52.626209 30542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:52.637878 30542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:52.639348 30542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:52.648923 30542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba578420ffe14b9ba9b063afa3cb8ae5: Generated new TSK 0
I20260928 15:07:52.649976 30542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:52.666338 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30456
W20260928 15:07:52.667577 30518 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:55513
W20260928 15:07:52.668329 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30456: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:52.668481 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30456
W20260928 15:07:52.685724 30441 connection.cc:535] client connection to 127.0.0.1:39593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39593): Connection reset by peer (error 104)
2026-09-28T15:07:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:52 dist-test-slave-dmhs krb5kdc[30554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:52 dist-test-slave-dmhs krb5kdc[30554](info): set up 2 sockets
Sep 28 15:07:52 dist-test-slave-dmhs krb5kdc[30554](info): commencing operation
krb5kdc: starting...
W20260928 15:07:53.749150 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:53 dist-test-slave-dmhs krb5kdc[30554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:54.012187 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:54.336218 30567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:54.336570 30567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:54.336738 30567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:54.344636 30567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:54.344774 30567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:54.344893 30567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:54.344982 30567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:54.345047 30567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:54.355253 30567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:54.356622 30567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:54.357903 30567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:54.369264 30572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:54.369216 30573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:54.369855 30575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:54.370872 30567 server_base.cc:1034] running on GCE node
I20260928 15:07:54.371855 30567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:54.373294 30567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:54.374536 30567 hybrid_clock.cc:648] HybridClock initialized: now 1790608074374488 us; error 58 us; skew 500 ppm
I20260928 15:07:54.374879 30567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:54 dist-test-slave-dmhs krb5kdc[30554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:54.382611 30567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:54.384049 30567 webserver.cc:460] Webserver started at http://127.0.0.1:35163/ using document root <none> and password file <none>
I20260928 15:07:54.384800 30567 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:54.384948 30567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:54.385252 30567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:54.388160 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "ea06129333494d6a84d22a90cbe593ce"
format_stamp: "Formatted at 2026-09-28 15:07:54 on dist-test-slave-dmhs"
server_key: "95647a0c668fcf648a61976fc138b724"
server_key_iv: "7d013aadc9feac92d0f0f6b6a6d67112"
server_key_version: "encryptionkey@0"
I20260928 15:07:54.389002 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "ea06129333494d6a84d22a90cbe593ce"
format_stamp: "Formatted at 2026-09-28 15:07:54 on dist-test-slave-dmhs"
server_key: "95647a0c668fcf648a61976fc138b724"
server_key_iv: "7d013aadc9feac92d0f0f6b6a6d67112"
server_key_version: "encryptionkey@0"
I20260928 15:07:54.393971 30567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:07:54.397284 30582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:54.398530 30567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:54.398712 30567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "ea06129333494d6a84d22a90cbe593ce"
format_stamp: "Formatted at 2026-09-28 15:07:54 on dist-test-slave-dmhs"
server_key: "95647a0c668fcf648a61976fc138b724"
server_key_iv: "7d013aadc9feac92d0f0f6b6a6d67112"
server_key_version: "encryptionkey@0"
I20260928 15:07:54.398918 30567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:54.404464 30567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:54.408152 30567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:54.408465 30567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:54.426621 30567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38661
I20260928 15:07:54.426674 30633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38661 every 8 connection(s)
I20260928 15:07:54.428021 30567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:54.431579 30634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:54.437285 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30567
I20260928 15:07:54.437485 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:54.438067 9267 external_mini_cluster.cc:1428] Setting key bf4e50264ca5e54ea04bbd45eb129d0e
I20260928 15:07:54.442565 30634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce: Bootstrap starting.
I20260928 15:07:54.446002 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:54.439849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:47427 (local address 127.0.0.1:38661)
0928 15:07:54.440802 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:54.440822 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:54.440864 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:07:54.442481 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:54.442503 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:54.442879 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:54.443182 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:54.444333 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:54.444342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:54.444354 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:54.444368 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:54.444429 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:54.444448 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:54.444622 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:54.444756 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 15:07:54.445005 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":487,"threads_started":1}
I20260928 15:07:54.446470 30634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:54.448017 30634 log.cc:824] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:54.450870 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:54.453465 30634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce: No bootstrap required, opened a new log
I20260928 15:07:54.459190 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:54.456697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:33505 (local address 127.0.0.1:38661)
0928 15:07:54.456884 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:54.456890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:54.457304 (+ 414us) server_negotiation.cc:407] Connection header received
0928 15:07:54.457430 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:54.457435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:54.457496 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:54.457598 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:54.458513 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:54.458519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:54.458529 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:54.458547 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:54.458598 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:54.458617 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:54.458749 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:54.458802 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 15:07:54.458874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:07:54.462468 30634 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea06129333494d6a84d22a90cbe593ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38661 } }
I20260928 15:07:54.462816 30634 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:54.462958 30634 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea06129333494d6a84d22a90cbe593ce, State: Initialized, Role: FOLLOWER
I20260928 15:07:54.463524 30634 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea06129333494d6a84d22a90cbe593ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38661 } }
I20260928 15:07:54.463757 30634 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:54.463907 30634 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:54.464082 30634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:54.466279 30634 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea06129333494d6a84d22a90cbe593ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38661 } }
I20260928 15:07:54.466759 30634 leader_election.cc:302] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea06129333494d6a84d22a90cbe593ce; no voters:
I20260928 15:07:54.467373 30634 leader_election.cc:288] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:54.467489 30644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:54.468149 30644 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [term 1 LEADER]: Becoming Leader. State: Replica: ea06129333494d6a84d22a90cbe593ce, State: Running, Role: LEADER
I20260928 15:07:54.468798 30644 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea06129333494d6a84d22a90cbe593ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38661 } }
I20260928 15:07:54.469739 30634 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:54.471848 30645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea06129333494d6a84d22a90cbe593ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea06129333494d6a84d22a90cbe593ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38661 } } }
I20260928 15:07:54.472184 30645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:54.473070 30651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:54.472924 30646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea06129333494d6a84d22a90cbe593ce. Latest consensus state: current_term: 1 leader_uuid: "ea06129333494d6a84d22a90cbe593ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea06129333494d6a84d22a90cbe593ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38661 } } }
I20260928 15:07:54.473219 30646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:54.478425 30651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:54.488199 30651 catalog_manager.cc:1347] Generated new cluster ID: a35b8e98577a4e6ba2c89e7927001ae3
I20260928 15:07:54.488358 30651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:54.507123 30651 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:54.507530 30651 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:54.520076 30651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea06129333494d6a84d22a90cbe593ce: Generated new TSK 0
I20260928 15:07:54.521032 30651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:54.525178 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30567
W20260928 15:07:54.526561 30629 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:47427
W20260928 15:07:54.527235 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30567: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:54.527387 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30567
2026-09-28T15:07:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:54 dist-test-slave-dmhs krb5kdc[30665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:54 dist-test-slave-dmhs krb5kdc[30665](info): set up 2 sockets
Sep 28 15:07:54 dist-test-slave-dmhs krb5kdc[30665](info): commencing operation
krb5kdc: starting...
W20260928 15:07:55.624940 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:55 dist-test-slave-dmhs krb5kdc[30665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:55.891089 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:56.212944 30678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:56.213227 30678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:56.213436 30678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:56.220470 30678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:56.220595 30678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:56.220741 30678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:56.220861 30678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:56.220947 30678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:56.230623 30678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:56.231895 30678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:56.233183 30678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:56.244750 30686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:56.244871 30684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:56.245054 30683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:56.245965 30678 server_base.cc:1034] running on GCE node
I20260928 15:07:56.246737 30678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:56.248071 30678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:56.249270 30678 hybrid_clock.cc:648] HybridClock initialized: now 1790608076249212 us; error 54 us; skew 500 ppm
I20260928 15:07:56.249586 30678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:56 dist-test-slave-dmhs krb5kdc[30665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:56.252676 30678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:56.254040 30678 webserver.cc:460] Webserver started at http://127.0.0.1:43577/ using document root <none> and password file <none>
I20260928 15:07:56.254715 30678 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:56.254837 30678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:56.255223 30678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:56.258353 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "5e733559d50a47fcb639eeceb47f16ae"
format_stamp: "Formatted at 2026-09-28 15:07:56 on dist-test-slave-dmhs"
server_key: "ea0d43ad233d850568e0dbe9e8bee961"
server_key_iv: "0cb3470f043e7598102544c23e40768b"
server_key_version: "encryptionkey@0"
I20260928 15:07:56.259128 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "5e733559d50a47fcb639eeceb47f16ae"
format_stamp: "Formatted at 2026-09-28 15:07:56 on dist-test-slave-dmhs"
server_key: "ea0d43ad233d850568e0dbe9e8bee961"
server_key_iv: "0cb3470f043e7598102544c23e40768b"
server_key_version: "encryptionkey@0"
I20260928 15:07:56.264200 30678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:07:56.267596 30693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:56.268983 30678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:56.269172 30678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "5e733559d50a47fcb639eeceb47f16ae"
format_stamp: "Formatted at 2026-09-28 15:07:56 on dist-test-slave-dmhs"
server_key: "ea0d43ad233d850568e0dbe9e8bee961"
server_key_iv: "0cb3470f043e7598102544c23e40768b"
server_key_version: "encryptionkey@0"
I20260928 15:07:56.269335 30678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:56.286427 30678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:56.290441 30678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:56.290742 30678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:56.308745 30678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46691
I20260928 15:07:56.308769 30744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46691 every 8 connection(s)
I20260928 15:07:56.310109 30678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:56.313824 30745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:56.316432 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30678
I20260928 15:07:56.316597 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:56.317159 9267 external_mini_cluster.cc:1428] Setting key c02769870917af2f42caf1c3c294c34b
I20260928 15:07:56.325155 30745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae: Bootstrap starting.
I20260928 15:07:56.328213 30745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:56.329239 30745 log.cc:824] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:07:56 dist-test-slave-dmhs krb5kdc[30665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:56.331817 30745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae: No bootstrap required, opened a new log
I20260928 15:07:56.338557 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:56.319059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:43161 (local address 127.0.0.1:46691)
0928 15:07:56.319793 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:56.319808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:56.319847 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:07:56.321540 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:56.321561 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:56.321960 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:56.322365 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:56.323829 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:56.325167 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:56.326478 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:56.327166 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:56.327362 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:56.331456 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:56.331463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:56.331476 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:56.331517 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:56.334538 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:56.335335 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:56.335341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:56.335346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:56.335496 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:56.335950 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:56.335953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:56.335955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:56.336469 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:56.336816 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:56.337307 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 15:07:56.337597 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":314,"threads_started":1}
I20260928 15:07:56.339737 30745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e733559d50a47fcb639eeceb47f16ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260928 15:07:56.340137 30745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:56.340287 30745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e733559d50a47fcb639eeceb47f16ae, State: Initialized, Role: FOLLOWER
I20260928 15:07:56.340971 30745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e733559d50a47fcb639eeceb47f16ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260928 15:07:56.341271 30745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:56.341426 30745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:56.341603 30745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:56.342829 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:56.344054 30745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e733559d50a47fcb639eeceb47f16ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260928 15:07:56.344573 30745 leader_election.cc:302] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e733559d50a47fcb639eeceb47f16ae; no voters:
I20260928 15:07:56.345217 30745 leader_election.cc:288] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:56.345350 30750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:56.348068 30750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [term 1 LEADER]: Becoming Leader. State: Replica: 5e733559d50a47fcb639eeceb47f16ae, State: Running, Role: LEADER
I20260928 15:07:56.348680 30750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e733559d50a47fcb639eeceb47f16ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260928 15:07:56.349161 30745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:56.353281 30754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e733559d50a47fcb639eeceb47f16ae. Latest consensus state: current_term: 1 leader_uuid: "5e733559d50a47fcb639eeceb47f16ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e733559d50a47fcb639eeceb47f16ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } } }
I20260928 15:07:56.353622 30754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:56.354173 30753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e733559d50a47fcb639eeceb47f16ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e733559d50a47fcb639eeceb47f16ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } } }
I20260928 15:07:56.354498 30753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:56.370669 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:56.347667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:37411 (local address 127.0.0.1:46691)
0928 15:07:56.347841 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:56.347846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:56.348904 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 15:07:56.349130 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:56.349135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:56.349212 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:56.349278 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:56.351737 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:56.353329 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:56.355841 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:56.356706 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:56.356855 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:56.360803 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:56.360823 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:56.360831 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:56.360885 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:56.365467 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:56.366723 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:56.366731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:56.366737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:56.366971 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:56.367662 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:56.367670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:56.367675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:56.368190 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:56.368487 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:56.369870 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0928 15:07:56.370127 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:07:56.373461 30764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:56.380716 30764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:56.388926 30764 catalog_manager.cc:1347] Generated new cluster ID: e96f1785447040229435736a1e6743b2
I20260928 15:07:56.389035 30764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:56.417439 30764 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:56.419330 30764 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:56.431864 30764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e733559d50a47fcb639eeceb47f16ae: Generated new TSK 0
I20260928 15:07:56.432860 30764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:56.442457 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30678
I20260928 15:07:56.443621 30740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:07:56.539930 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30678
W20260928 15:07:56.557421 30663 connection.cc:535] client connection to 127.0.0.1:46691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46691): Connection reset by peer (error 104)
2026-09-28T15:07:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:56 dist-test-slave-dmhs krb5kdc[30778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:56 dist-test-slave-dmhs krb5kdc[30778](info): set up 2 sockets
Sep 28 15:07:56 dist-test-slave-dmhs krb5kdc[30778](info): commencing operation
krb5kdc: starting...
W20260928 15:07:57.640987 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:57 dist-test-slave-dmhs krb5kdc[30778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:57.901368 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:58.228914 30791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:58.229259 30791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:58.229435 30791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:58.236861 30791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:58.237028 30791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:58.237136 30791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:58.237247 30791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:58.237326 30791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:58.247344 30791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:07:58.248759 30791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:58.250137 30791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:58.261981 30797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:58.262044 30799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:58.262077 30796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:58.262795 30791 server_base.cc:1034] running on GCE node
I20260928 15:07:58.263520 30791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:58.264948 30791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:58.266155 30791 hybrid_clock.cc:648] HybridClock initialized: now 1790608078266109 us; error 39 us; skew 500 ppm
I20260928 15:07:58.266494 30791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:07:58 dist-test-slave-dmhs krb5kdc[30778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:58.269629 30791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:58.271112 30791 webserver.cc:460] Webserver started at http://127.0.0.1:38233/ using document root <none> and password file <none>
I20260928 15:07:58.271756 30791 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:58.271888 30791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:58.272200 30791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:58.275151 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "9fdd1647c29141f49ca6338c69bbcc38"
format_stamp: "Formatted at 2026-09-28 15:07:58 on dist-test-slave-dmhs"
server_key: "73a5c703bcfa4c250ec8eebc2b665abe"
server_key_iv: "7bad0b434096083befcd435dce4c762e"
server_key_version: "encryptionkey@0"
I20260928 15:07:58.275913 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "9fdd1647c29141f49ca6338c69bbcc38"
format_stamp: "Formatted at 2026-09-28 15:07:58 on dist-test-slave-dmhs"
server_key: "73a5c703bcfa4c250ec8eebc2b665abe"
server_key_iv: "7bad0b434096083befcd435dce4c762e"
server_key_version: "encryptionkey@0"
I20260928 15:07:58.281195 30791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 15:07:58.284588 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:58.285859 30791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:07:58.286051 30791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "9fdd1647c29141f49ca6338c69bbcc38"
format_stamp: "Formatted at 2026-09-28 15:07:58 on dist-test-slave-dmhs"
server_key: "73a5c703bcfa4c250ec8eebc2b665abe"
server_key_iv: "7bad0b434096083befcd435dce4c762e"
server_key_version: "encryptionkey@0"
I20260928 15:07:58.286248 30791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:58.308025 30791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:58.312181 30791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:58.312515 30791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:58.331545 30791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34391
I20260928 15:07:58.331575 30857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34391 every 8 connection(s)
I20260928 15:07:58.332939 30791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:58.335654 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30791
I20260928 15:07:58.335845 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:07:58.336398 9267 external_mini_cluster.cc:1428] Setting key 598fed2996d0660f24e2c496014c7094
I20260928 15:07:58.336751 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:58.350389 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:58.338155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:47175 (local address 127.0.0.1:34391)
0928 15:07:58.338929 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:58.338941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:58.338987 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:07:58.341333 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:58.341354 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:58.341776 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:58.342122 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:58.342722 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:58.344031 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:58.345920 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:58.346571 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:58.346709 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:58.348101 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:58.348112 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:58.348125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:58.348731 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:58.348814 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:58.348837 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:58.348959 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:58.349275 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:07:58.349453 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":344,"threads_started":1}
I20260928 15:07:58.350430 30858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38: Bootstrap starting.
I20260928 15:07:58.353917 30858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:58.354259 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:58.354961 30858 log.cc:824] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:58.357620 30858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38: No bootstrap required, opened a new log
I20260928 15:07:58.365782 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:58.359090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:59003 (local address 127.0.0.1:34391)
0928 15:07:58.359295 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:58.359300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:58.359581 (+ 281us) server_negotiation.cc:407] Connection header received
0928 15:07:58.359759 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:58.359762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:58.359825 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:58.359901 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:58.360528 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:58.361320 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:58.363289 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:58.363730 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:58.363833 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:58.365013 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:58.365018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:58.365021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:58.365040 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:58.365075 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:58.365091 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:58.365174 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:58.365380 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 15:07:58.365422 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:07:58.367206 30858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd1647c29141f49ca6338c69bbcc38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34391 } }
I20260928 15:07:58.367573 30858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:58.367686 30858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdd1647c29141f49ca6338c69bbcc38, State: Initialized, Role: FOLLOWER
I20260928 15:07:58.368238 30858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd1647c29141f49ca6338c69bbcc38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34391 } }
I20260928 15:07:58.368549 30858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:58.368687 30858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:58.368868 30858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:58.370965 30858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd1647c29141f49ca6338c69bbcc38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34391 } }
I20260928 15:07:58.371450 30858 leader_election.cc:302] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fdd1647c29141f49ca6338c69bbcc38; no voters:
I20260928 15:07:58.372001 30858 leader_election.cc:288] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:58.372128 30868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:58.373610 30868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [term 1 LEADER]: Becoming Leader. State: Replica: 9fdd1647c29141f49ca6338c69bbcc38, State: Running, Role: LEADER
I20260928 15:07:58.374140 30868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd1647c29141f49ca6338c69bbcc38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34391 } }
I20260928 15:07:58.374285 30858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:58.377118 30869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fdd1647c29141f49ca6338c69bbcc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd1647c29141f49ca6338c69bbcc38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34391 } } }
I20260928 15:07:58.377308 30870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdd1647c29141f49ca6338c69bbcc38. Latest consensus state: current_term: 1 leader_uuid: "9fdd1647c29141f49ca6338c69bbcc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd1647c29141f49ca6338c69bbcc38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34391 } } }
I20260928 15:07:58.377564 30870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:58.377564 30869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:58.379401 30878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:58.384387 30878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:58.393404 30878 catalog_manager.cc:1347] Generated new cluster ID: f036faed1f9f48ad8acdb65934c75e2b
I20260928 15:07:58.393532 30878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:58.405632 30878 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:58.406906 30878 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:58.416533 30878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fdd1647c29141f49ca6338c69bbcc38: Generated new TSK 0
I20260928 15:07:58.417344 30878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:58.427776 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30791
W20260928 15:07:58.428819 30853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:47175
W20260928 15:07:58.429553 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30791: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:07:58.429692 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30791
W20260928 15:07:58.446455 30776 connection.cc:535] client connection to 127.0.0.1:34391 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34391): Connection reset by peer (error 104)
2026-09-28T15:07:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:58 dist-test-slave-dmhs krb5kdc[30889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:58 dist-test-slave-dmhs krb5kdc[30889](info): set up 2 sockets
Sep 28 15:07:58 dist-test-slave-dmhs krb5kdc[30889](info): commencing operation
krb5kdc: starting...
W20260928 15:07:59.521003 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:07:59 dist-test-slave-dmhs krb5kdc[30889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:59.783751 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:00.110725 30902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:00.111089 30902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:00.111267 30902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:00.118517 30902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:00.118687 30902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:00.118795 30902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:00.118940 30902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:00.119027 30902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:00.128997 30902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:00.130373 30902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:00.131726 30902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:00.143427 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:00.143428 30908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:00.143469 30910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:00.144212 30902 server_base.cc:1034] running on GCE node
I20260928 15:08:00.144948 30902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:00.146201 30902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:00.147425 30902 hybrid_clock.cc:648] HybridClock initialized: now 1790608080147383 us; error 29 us; skew 500 ppm
I20260928 15:08:00.147809 30902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:00 dist-test-slave-dmhs krb5kdc[30889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:00.156222 30902 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:08:00.157866 30902 webserver.cc:460] Webserver started at http://127.0.0.1:46089/ using document root <none> and password file <none>
I20260928 15:08:00.158588 30902 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:00.158738 30902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:00.159057 30902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:00.162191 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c"
format_stamp: "Formatted at 2026-09-28 15:08:00 on dist-test-slave-dmhs"
server_key: "3dc4ce2496269ac5f87749767d6f9697"
server_key_iv: "fe5380f191ed70a6cbafe516de48f8fa"
server_key_version: "encryptionkey@0"
I20260928 15:08:00.162987 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c"
format_stamp: "Formatted at 2026-09-28 15:08:00 on dist-test-slave-dmhs"
server_key: "3dc4ce2496269ac5f87749767d6f9697"
server_key_iv: "fe5380f191ed70a6cbafe516de48f8fa"
server_key_version: "encryptionkey@0"
I20260928 15:08:00.168219 30902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:08:00.171856 30917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:00.173281 30902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:08:00.173476 30902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c"
format_stamp: "Formatted at 2026-09-28 15:08:00 on dist-test-slave-dmhs"
server_key: "3dc4ce2496269ac5f87749767d6f9697"
server_key_iv: "fe5380f191ed70a6cbafe516de48f8fa"
server_key_version: "encryptionkey@0"
I20260928 15:08:00.173671 30902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:00.179347 30902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:00.183712 30902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:00.184024 30902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:00.202011 30902 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35041
I20260928 15:08:00.202061 30968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35041 every 8 connection(s)
I20260928 15:08:00.203434 30902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:00.207186 30969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:00.209360 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 30902
I20260928 15:08:00.209550 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:00.210071 9267 external_mini_cluster.cc:1428] Setting key 17eee40ebc0cb0efd25d635c5745bcbd
I20260928 15:08:00.217561 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:00.211767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:50653 (local address 127.0.0.1:35041)
0928 15:08:00.212677 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:00.212688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:00.212733 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:08:00.214211 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:00.214235 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:00.214572 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:00.214880 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:00.215936 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:00.215942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:00.215954 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:00.215967 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:00.216022 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:00.216041 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:00.216208 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:00.216417 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:08:00.216647 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":483,"threads_started":1}
I20260928 15:08:00.221323 30969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c: Bootstrap starting.
I20260928 15:08:00.221340 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:00.224329 30969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:00.225397 30969 log.cc:824] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:00.228168 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:00.225640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:55297 (local address 127.0.0.1:35041)
0928 15:08:00.225834 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:00.225841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:00.226217 (+ 376us) server_negotiation.cc:407] Connection header received
0928 15:08:00.226338 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:00.226346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:00.226409 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:00.226531 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:00.227445 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:00.227449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:00.227452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:00.227465 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:00.227502 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:00.227516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:00.227602 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:00.227795 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:08:00.227846 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:08:00.228787 30969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c: No bootstrap required, opened a new log
I20260928 15:08:00.235916 30969 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 15:08:00.236353 30969 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:00.236464 30969 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccec3fbfa1144eb5b4baed5ccbafbc5c, State: Initialized, Role: FOLLOWER
I20260928 15:08:00.237061 30969 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 15:08:00.237324 30969 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:00.237493 30969 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:00.237689 30969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:00.239969 30969 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 15:08:00.240568 30969 leader_election.cc:302] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccec3fbfa1144eb5b4baed5ccbafbc5c; no voters:
I20260928 15:08:00.241107 30969 leader_election.cc:288] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:00.241271 30979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:00.241938 30979 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [term 1 LEADER]: Becoming Leader. State: Replica: ccec3fbfa1144eb5b4baed5ccbafbc5c, State: Running, Role: LEADER
I20260928 15:08:00.242447 30979 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 15:08:00.243460 30969 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:00.245566 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccec3fbfa1144eb5b4baed5ccbafbc5c. Latest consensus state: current_term: 1 leader_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } } }
I20260928 15:08:00.246037 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:00.246951 30980 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccec3fbfa1144eb5b4baed5ccbafbc5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } } }
I20260928 15:08:00.247295 30980 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:00.249251 30986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:00.254648 30986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:00.263242 30986 catalog_manager.cc:1347] Generated new cluster ID: 1a215f15ab394f1c8ae3fe1a597d2705
I20260928 15:08:00.263372 30986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:00.272208 30986 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:00.272712 30986 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:00.284818 30986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccec3fbfa1144eb5b4baed5ccbafbc5c: Generated new TSK 0
I20260928 15:08:00.285962 30986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:00.290864 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30902
W20260928 15:08:00.292006 30964 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:50653
W20260928 15:08:00.292877 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 30902: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:08:00.293051 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 30902
2026-09-28T15:08:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:00 dist-test-slave-dmhs krb5kdc[31000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:00 dist-test-slave-dmhs krb5kdc[31000](info): set up 2 sockets
Sep 28 15:08:00 dist-test-slave-dmhs krb5kdc[31000](info): commencing operation
krb5kdc: starting...
W20260928 15:08:01.397284 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:01 dist-test-slave-dmhs krb5kdc[31000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:01.653584 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:35061
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:01.975737 31013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:01.976083 31013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:01.976306 31013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:01.983578 31013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:01.983732 31013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:01.983852 31013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:01.983954 31013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:01.984037 31013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:01.994109 31013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:35061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:01.995414 31013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:01.996688 31013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:02.008345 31019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:02.008378 31018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:02.008543 31021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:02.009519 31013 server_base.cc:1034] running on GCE node
I20260928 15:08:02.010227 31013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:02.011518 31013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:02.012706 31013 hybrid_clock.cc:648] HybridClock initialized: now 1790608082012659 us; error 42 us; skew 500 ppm
I20260928 15:08:02.013052 31013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:02 dist-test-slave-dmhs krb5kdc[31000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608082, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:02.016048 31013 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:02.017436 31013 webserver.cc:460] Webserver started at http://127.9.12.254:35131/ using document root <none> and password file <none>
I20260928 15:08:02.018186 31013 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:02.018298 31013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:02.018543 31013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:02.021472 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "acf92ff869004d53a1438dbafa0b8dd9"
format_stamp: "Formatted at 2026-09-28 15:08:02 on dist-test-slave-dmhs"
server_key: "822145b19074a0c521fff4db68db067d"
server_key_iv: "564eeb6f35ea04528a16c8ab120618cf"
server_key_version: "encryptionkey@0"
I20260928 15:08:02.022156 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "acf92ff869004d53a1438dbafa0b8dd9"
format_stamp: "Formatted at 2026-09-28 15:08:02 on dist-test-slave-dmhs"
server_key: "822145b19074a0c521fff4db68db067d"
server_key_iv: "564eeb6f35ea04528a16c8ab120618cf"
server_key_version: "encryptionkey@0"
I20260928 15:08:02.027117 31013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:08:02.030372 31028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:02.031749 31013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:08:02.031940 31013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "acf92ff869004d53a1438dbafa0b8dd9"
format_stamp: "Formatted at 2026-09-28 15:08:02 on dist-test-slave-dmhs"
server_key: "822145b19074a0c521fff4db68db067d"
server_key_iv: "564eeb6f35ea04528a16c8ab120618cf"
server_key_version: "encryptionkey@0"
I20260928 15:08:02.032141 31013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:02.053042 31013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:02.056882 31013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:02.057196 31013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:02.075641 31013 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:35061
I20260928 15:08:02.075667 31079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:35061 every 8 connection(s)
I20260928 15:08:02.077158 31013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:02.079406 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31013
I20260928 15:08:02.079613 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:02.080063 9267 external_mini_cluster.cc:1428] Setting key a80b6f9bba5e8aef0bd5def142f12c57
I20260928 15:08:02.081332 31080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:08:02 dist-test-slave-dmhs krb5kdc[31000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:08:02.094722 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9: Bootstrap starting.
I20260928 15:08:02.098694 31080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:02.100041 31080 log.cc:824] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:02.102862 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9: No bootstrap required, opened a new log
I20260928 15:08:02.103951 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:02.081878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46167 (local address 127.9.12.254:35061)
0928 15:08:02.082638 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:02.082655 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:02.082703 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:08:02.084693 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:02.084718 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:02.085214 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:02.085635 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:02.087089 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:02.089151 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:02.090392 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:02.091441 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:02.091632 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:02.095800 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:02.095809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:02.095825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:02.095871 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:02.099772 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:02.100662 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:02.100668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:02.100674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:02.100841 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:02.101319 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:02.101323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:02.101325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:02.101829 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:02.102154 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:02.102684 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 15:08:02.103035 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":326,"threads_started":1}
I20260928 15:08:02.107673 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:02.110632 31080 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf92ff869004d53a1438dbafa0b8dd9" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35061 } }
I20260928 15:08:02.110988 31080 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:02.111115 31080 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf92ff869004d53a1438dbafa0b8dd9, State: Initialized, Role: FOLLOWER
I20260928 15:08:02.111634 31080 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf92ff869004d53a1438dbafa0b8dd9" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35061 } }
I20260928 15:08:02.111883 31080 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:02.112037 31080 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:02.112236 31080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:02.114377 31080 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf92ff869004d53a1438dbafa0b8dd9" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35061 } }
I20260928 15:08:02.114872 31080 leader_election.cc:302] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf92ff869004d53a1438dbafa0b8dd9; no voters:
I20260928 15:08:02.115602 31080 leader_election.cc:288] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:02.115686 31090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:02.118208 31080 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:02.122264 31090 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [term 1 LEADER]: Becoming Leader. State: Replica: acf92ff869004d53a1438dbafa0b8dd9, State: Running, Role: LEADER
I20260928 15:08:02.123034 31090 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf92ff869004d53a1438dbafa0b8dd9" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35061 } }
I20260928 15:08:02.126389 31092 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf92ff869004d53a1438dbafa0b8dd9. Latest consensus state: current_term: 1 leader_uuid: "acf92ff869004d53a1438dbafa0b8dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf92ff869004d53a1438dbafa0b8dd9" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35061 } } }
I20260928 15:08:02.126948 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:02.111971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35551 (local address 127.9.12.254:35061)
0928 15:08:02.112147 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:02.112153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:02.112749 (+ 596us) server_negotiation.cc:407] Connection header received
0928 15:08:02.112933 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:02.112941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:02.113022 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:02.113138 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:02.114540 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:02.115395 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:02.117045 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:02.117574 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:02.117690 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:02.120426 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:02.120432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:02.120437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:02.120479 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:02.123507 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:02.124334 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:02.124339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:02.124342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:02.124467 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:02.125008 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:02.125014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:02.125018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:02.125435 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:02.125769 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:02.126311 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 15:08:02.126524 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:08:02.126957 31092 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:02.127148 31091 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf92ff869004d53a1438dbafa0b8dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf92ff869004d53a1438dbafa0b8dd9" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35061 } } }
I20260928 15:08:02.127826 31091 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:02.131742 31099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:02.138262 31099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:02.147286 31099 catalog_manager.cc:1347] Generated new cluster ID: 070b417a4ad042cd9f83752975f6aebb
I20260928 15:08:02.147400 31099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:02.164536 31099 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:02.165760 31099 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:02.174052 31099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf92ff869004d53a1438dbafa0b8dd9: Generated new TSK 0
I20260928 15:08:02.174851 31099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:02.182466 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31013
I20260928 15:08:02.183851 31075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:02.301880 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31013
W20260928 15:08:02.319370 30998 connection.cc:535] client connection to 127.9.12.254:35061 recv error: Network error: failed to read from TLS socket (remote: 127.9.12.254:35061): Connection reset by peer (error 104)
2026-09-28T15:08:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:02 dist-test-slave-dmhs krb5kdc[31113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:02 dist-test-slave-dmhs krb5kdc[31113](info): set up 2 sockets
Sep 28 15:08:02 dist-test-slave-dmhs krb5kdc[31113](info): commencing operation
krb5kdc: starting...
W20260928 15:08:03.384738 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:03 dist-test-slave-dmhs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:03.636612 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43107
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:43107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:03.963186 31126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:03.963440 31126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:03.963542 31126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:03.970551 31126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:03.970656 31126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:03.970697 31126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:03.970754 31126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:03.970788 31126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:03.980152 31126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:43107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:03.981397 31126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:03.982620 31126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:03.994715 31132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:03.995078 31131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:03.995157 31134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:03.995317 31126 server_base.cc:1034] running on GCE node
I20260928 15:08:03.996234 31126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:03.997624 31126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:03.998864 31126 hybrid_clock.cc:648] HybridClock initialized: now 1790608083998823 us; error 53 us; skew 500 ppm
I20260928 15:08:03.999195 31126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:03 dist-test-slave-dmhs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:04.002285 31126 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:04.003695 31126 webserver.cc:460] Webserver started at http://127.9.12.254:35221/ using document root <none> and password file <none>
I20260928 15:08:04.004415 31126 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:04.004558 31126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:04.004858 31126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:04.007781 31126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "416ab23b49f34c4db94bb105958fc141"
format_stamp: "Formatted at 2026-09-28 15:08:04 on dist-test-slave-dmhs"
server_key: "4a96a46ffafa143150e670bd09478572"
server_key_iv: "3c8615de3120172984a2126ecc82e117"
server_key_version: "encryptionkey@0"
I20260928 15:08:04.008613 31126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "416ab23b49f34c4db94bb105958fc141"
format_stamp: "Formatted at 2026-09-28 15:08:04 on dist-test-slave-dmhs"
server_key: "4a96a46ffafa143150e670bd09478572"
server_key_iv: "3c8615de3120172984a2126ecc82e117"
server_key_version: "encryptionkey@0"
I20260928 15:08:04.013717 31126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:08:04.017248 31141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:04.018532 31126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:08:04.018716 31126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "416ab23b49f34c4db94bb105958fc141"
format_stamp: "Formatted at 2026-09-28 15:08:04 on dist-test-slave-dmhs"
server_key: "4a96a46ffafa143150e670bd09478572"
server_key_iv: "3c8615de3120172984a2126ecc82e117"
server_key_version: "encryptionkey@0"
I20260928 15:08:04.018913 31126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:04.030052 31126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:04.034346 31126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:04.034641 31126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:04.052321 31126 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:43107
I20260928 15:08:04.052381 31192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:43107 every 8 connection(s)
I20260928 15:08:04.053738 31126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:04.057389 31193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:04.061260 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31126
I20260928 15:08:04.061447 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:04.061975 9267 external_mini_cluster.cc:1428] Setting key 60bc8e45d0d03e1b7acc5a97236daf58
I20260928 15:08:04.069587 31193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141: Bootstrap starting.
I20260928 15:08:04.073112 31193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:04.074254 31193 log.cc:824] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:04.074574 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:04.063610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45029 (local address 127.9.12.254:43107)
0928 15:08:04.064351 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:04.064363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:04.064400 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:08:04.065820 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:04.065840 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:04.066197 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:04.066483 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:04.067077 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:04.068410 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:04.070239 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:04.070875 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:04.071005 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:04.072354 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:04.072364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:04.072376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:04.072879 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:04.072942 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:04.072970 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:04.073079 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:04.073416 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:08:04.073612 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":311,"threads_started":1}
I20260928 15:08:04.078117 31193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141: No bootstrap required, opened a new log
I20260928 15:08:04.078356 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:04.088642 31193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416ab23b49f34c4db94bb105958fc141" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43107 } }
I20260928 15:08:04.089066 31193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:04.089193 31193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416ab23b49f34c4db94bb105958fc141, State: Initialized, Role: FOLLOWER
I20260928 15:08:04.089969 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:04.083618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35615 (local address 127.9.12.254:43107)
0928 15:08:04.083755 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:04.083759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:04.084094 (+ 335us) server_negotiation.cc:407] Connection header received
0928 15:08:04.084303 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:04.084313 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:04.084372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:04.084448 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:04.085003 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:04.085724 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:04.087535 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:04.087964 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:04.088063 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:04.089200 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:04.089206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:04.089214 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:04.089229 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:04.089276 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:04.089308 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:04.089458 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:04.089652 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 15:08:04.089702 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 15:08:04.089921 31193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416ab23b49f34c4db94bb105958fc141" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43107 } }
I20260928 15:08:04.090209 31193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:04.090361 31193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:04.090579 31193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:04.093153 31193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416ab23b49f34c4db94bb105958fc141" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43107 } }
I20260928 15:08:04.093576 31193 leader_election.cc:302] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416ab23b49f34c4db94bb105958fc141; no voters:
I20260928 15:08:04.094099 31193 leader_election.cc:288] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:04.094234 31203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:04.094935 31203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [term 1 LEADER]: Becoming Leader. State: Replica: 416ab23b49f34c4db94bb105958fc141, State: Running, Role: LEADER
I20260928 15:08:04.095453 31203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416ab23b49f34c4db94bb105958fc141" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43107 } }
I20260928 15:08:04.096179 31193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:04.098055 31205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416ab23b49f34c4db94bb105958fc141. Latest consensus state: current_term: 1 leader_uuid: "416ab23b49f34c4db94bb105958fc141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416ab23b49f34c4db94bb105958fc141" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43107 } } }
I20260928 15:08:04.098524 31205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:04.099325 31211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:04.101375 31204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416ab23b49f34c4db94bb105958fc141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416ab23b49f34c4db94bb105958fc141" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 43107 } } }
I20260928 15:08:04.101725 31204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:04.109450 31211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:04.117952 31211 catalog_manager.cc:1347] Generated new cluster ID: d073e4fee8c14dde98a55f18572591c5
I20260928 15:08:04.118074 31211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:04.126534 31211 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:04.127699 31211 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:04.135640 31211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416ab23b49f34c4db94bb105958fc141: Generated new TSK 0
I20260928 15:08:04.136536 31211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:04.150622 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31126
W20260928 15:08:04.151961 31188 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45029
W20260928 15:08:04.152709 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31126: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:08:04.152870 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31126
W20260928 15:08:04.169313 31111 connection.cc:535] client connection to 127.9.12.254:43107 recv error: Network error: failed to read from TLS socket (remote: 127.9.12.254:43107): Connection reset by peer (error 104)
2026-09-28T15:08:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:04 dist-test-slave-dmhs krb5kdc[31224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:04 dist-test-slave-dmhs krb5kdc[31224](info): set up 2 sockets
Sep 28 15:08:04 dist-test-slave-dmhs krb5kdc[31224](info): commencing operation
krb5kdc: starting...
W20260928 15:08:05.260777 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:05 dist-test-slave-dmhs krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:05.512159 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40551
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:05.836024 31237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:05.836385 31237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:05.836546 31237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:05.843739 31237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:05.843905 31237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:05.844033 31237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:05.844136 31237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:05.844224 31237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:05.853962 31237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:05.855293 31237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:05.856652 31237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:05.868989 31242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:05.868954 31243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:05.869040 31245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:05.870206 31237 server_base.cc:1034] running on GCE node
I20260928 15:08:05.870995 31237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:05.872490 31237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:05.873744 31237 hybrid_clock.cc:648] HybridClock initialized: now 1790608085873683 us; error 47 us; skew 500 ppm
I20260928 15:08:05.874114 31237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:05 dist-test-slave-dmhs krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608085, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:05.882794 31237 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:05.884186 31237 webserver.cc:460] Webserver started at http://127.9.12.254:44923/ using document root <none> and password file <none>
I20260928 15:08:05.884934 31237 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:05.885087 31237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:05.885411 31237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:05.888545 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "faf22646809e4bdd89f4e2080cd6f9af"
format_stamp: "Formatted at 2026-09-28 15:08:05 on dist-test-slave-dmhs"
server_key: "e96222b73df8245e3a3ef4d191155e0b"
server_key_iv: "2e32bba625c380cb7f1bf80833c55590"
server_key_version: "encryptionkey@0"
I20260928 15:08:05.889325 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "faf22646809e4bdd89f4e2080cd6f9af"
format_stamp: "Formatted at 2026-09-28 15:08:05 on dist-test-slave-dmhs"
server_key: "e96222b73df8245e3a3ef4d191155e0b"
server_key_iv: "2e32bba625c380cb7f1bf80833c55590"
server_key_version: "encryptionkey@0"
I20260928 15:08:05.894392 31237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:08:05.897742 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:05.898922 31237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:08:05.899106 31237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "faf22646809e4bdd89f4e2080cd6f9af"
format_stamp: "Formatted at 2026-09-28 15:08:05 on dist-test-slave-dmhs"
server_key: "e96222b73df8245e3a3ef4d191155e0b"
server_key_iv: "2e32bba625c380cb7f1bf80833c55590"
server_key_version: "encryptionkey@0"
I20260928 15:08:05.899290 31237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:05.905064 31237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:05.908952 31237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:05.909271 31237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:05.928092 31237 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:40551
I20260928 15:08:05.928131 31303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:40551 every 8 connection(s)
I20260928 15:08:05.929533 31237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:05.933593 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:05.936527 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31237
I20260928 15:08:05.936693 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:05.937129 9267 external_mini_cluster.cc:1428] Setting key c348089d17d20e741014defbbb3f7421
I20260928 15:08:05.944917 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:05.939097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58647 (local address 127.9.12.254:40551)
0928 15:08:05.939881 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:05.939893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:05.939931 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:08:05.941633 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:05.941654 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:05.941994 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:05.942309 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:05.943319 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:05.943325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:05.943336 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:05.943356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:05.943414 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:05.943435 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:05.943605 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:05.943708 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 15:08:05.943925 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":359,"threads_started":1}
I20260928 15:08:05.945564 31304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af: Bootstrap starting.
I20260928 15:08:05.948325 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:05.948823 31304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:05.950001 31304 log.cc:824] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:05.953176 31304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af: No bootstrap required, opened a new log
I20260928 15:08:05.958724 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:05.953446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47715 (local address 127.9.12.254:40551)
0928 15:08:05.953665 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:05.953674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:05.956790 (+ 3116us) server_negotiation.cc:407] Connection header received
0928 15:08:05.956949 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:05.956953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:05.957026 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:05.957102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:05.958026 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:05.958030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:05.958033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:05.958049 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:05.958084 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:05.958100 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:05.958193 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:05.958357 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 15:08:05.958398 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:08:05.961216 31304 raft_consensus.cc:348] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf22646809e4bdd89f4e2080cd6f9af" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40551 } }
I20260928 15:08:05.961604 31304 raft_consensus.cc:374] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:05.961738 31304 raft_consensus.cc:729] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf22646809e4bdd89f4e2080cd6f9af, State: Initialized, Role: FOLLOWER
I20260928 15:08:05.962311 31304 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf22646809e4bdd89f4e2080cd6f9af" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40551 } }
I20260928 15:08:05.962517 31304 raft_consensus.cc:388] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:05.962613 31304 raft_consensus.cc:482] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:05.962745 31304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:05.964767 31304 raft_consensus.cc:504] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf22646809e4bdd89f4e2080cd6f9af" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40551 } }
I20260928 15:08:05.965160 31304 leader_election.cc:302] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf22646809e4bdd89f4e2080cd6f9af; no voters:
I20260928 15:08:05.965631 31304 leader_election.cc:288] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:05.965759 31314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:05.966440 31314 raft_consensus.cc:686] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [term 1 LEADER]: Becoming Leader. State: Replica: faf22646809e4bdd89f4e2080cd6f9af, State: Running, Role: LEADER
I20260928 15:08:05.966974 31314 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf22646809e4bdd89f4e2080cd6f9af" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40551 } }
I20260928 15:08:05.967729 31304 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:05.970016 31315 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faf22646809e4bdd89f4e2080cd6f9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf22646809e4bdd89f4e2080cd6f9af" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40551 } } }
I20260928 15:08:05.970458 31315 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:05.971050 31316 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf22646809e4bdd89f4e2080cd6f9af. Latest consensus state: current_term: 1 leader_uuid: "faf22646809e4bdd89f4e2080cd6f9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf22646809e4bdd89f4e2080cd6f9af" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40551 } } }
I20260928 15:08:05.971432 31316 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:05.972766 31323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:05.978116 31323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:05.986657 31323 catalog_manager.cc:1347] Generated new cluster ID: 2ea76cabfdb44791b6a5eab320ae5ab7
I20260928 15:08:05.986930 31323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:06.015894 31323 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:06.016413 31323 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:06.026158 31323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faf22646809e4bdd89f4e2080cd6f9af: Generated new TSK 0
I20260928 15:08:06.027223 31323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:06.045246 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31237
W20260928 15:08:06.046375 31299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58647
W20260928 15:08:06.047134 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:08:06.047272 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31237
2026-09-28T15:08:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:06 dist-test-slave-dmhs krb5kdc[31335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:06 dist-test-slave-dmhs krb5kdc[31335](info): set up 2 sockets
Sep 28 15:08:06 dist-test-slave-dmhs krb5kdc[31335](info): commencing operation
krb5kdc: starting...
W20260928 15:08:07.129022 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:07 dist-test-slave-dmhs krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:07.383270 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45481
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:45481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:07.706334 31348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:07.706669 31348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:07.706859 31348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:07.713934 31348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:07.714094 31348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:07.714190 31348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:07.714299 31348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:07.714381 31348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:07.723851 31348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:45481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:45481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:07.725167 31348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:07.726460 31348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:07.738445 31353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:07.738412 31356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:07.738399 31354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:07.739071 31348 server_base.cc:1034] running on GCE node
I20260928 15:08:07.739897 31348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:07.741334 31348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:07.742566 31348 hybrid_clock.cc:648] HybridClock initialized: now 1790608087742499 us; error 57 us; skew 500 ppm
I20260928 15:08:07.742941 31348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:07 dist-test-slave-dmhs krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608087, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:07.746271 31348 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:07.747819 31348 webserver.cc:460] Webserver started at http://127.9.12.254:33127/ using document root <none> and password file <none>
I20260928 15:08:07.748588 31348 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:07.748732 31348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:07.749039 31348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:07.752009 31348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "821f593ade0a4a048ab73457258d39db"
format_stamp: "Formatted at 2026-09-28 15:08:07 on dist-test-slave-dmhs"
server_key: "817976faf85ecc8b6007508221ff1e20"
server_key_iv: "7de26edb43e3d6e7082b8776318a7525"
server_key_version: "encryptionkey@0"
I20260928 15:08:07.752806 31348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "821f593ade0a4a048ab73457258d39db"
format_stamp: "Formatted at 2026-09-28 15:08:07 on dist-test-slave-dmhs"
server_key: "817976faf85ecc8b6007508221ff1e20"
server_key_iv: "7de26edb43e3d6e7082b8776318a7525"
server_key_version: "encryptionkey@0"
I20260928 15:08:07.757844 31348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:08:07.761195 31363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:07.762457 31348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:08:07.762648 31348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "821f593ade0a4a048ab73457258d39db"
format_stamp: "Formatted at 2026-09-28 15:08:07 on dist-test-slave-dmhs"
server_key: "817976faf85ecc8b6007508221ff1e20"
server_key_iv: "7de26edb43e3d6e7082b8776318a7525"
server_key_version: "encryptionkey@0"
I20260928 15:08:07.762848 31348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:07.795563 31348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:07.800073 31348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:07.800441 31348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:07.819512 31348 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:45481
I20260928 15:08:07.819561 31414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:45481 every 8 connection(s)
I20260928 15:08:07.820923 31348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:07.824730 31415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:07.826766 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31348
I20260928 15:08:07.826982 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:07.827527 9267 external_mini_cluster.cc:1428] Setting key ab535cd0d274e6a14a2d7aa80bd5340a
I20260928 15:08:07.839305 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db: Bootstrap starting.
Sep 28 15:08:07 dist-test-slave-dmhs krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:08:07.843173 31415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:07.844224 31415 log.cc:824] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:07.847240 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db: No bootstrap required, opened a new log
I20260928 15:08:07.851269 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:07.829390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42917 (local address 127.9.12.254:45481)
0928 15:08:07.830209 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:07.830221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:07.830263 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:08:07.831846 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:07.831867 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:07.832255 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:07.832635 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:07.834397 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:07.835702 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:07.837589 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:07.838484 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:07.838690 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:07.843670 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:07.843678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:07.843693 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:07.843727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:07.846980 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:07.847817 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:07.847824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:07.847830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:07.848010 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:07.848480 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:07.848483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:07.848485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:07.849031 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:07.849387 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:07.849846 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 15:08:07.850278 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":387,"threads_started":1}
I20260928 15:08:07.855052 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:07.855492 31415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f593ade0a4a048ab73457258d39db" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45481 } }
I20260928 15:08:07.855870 31415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:07.855976 31415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821f593ade0a4a048ab73457258d39db, State: Initialized, Role: FOLLOWER
I20260928 15:08:07.856578 31415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f593ade0a4a048ab73457258d39db" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45481 } }
I20260928 15:08:07.856838 31415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:07.856982 31415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:07.857165 31415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:07.859704 31415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f593ade0a4a048ab73457258d39db" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45481 } }
I20260928 15:08:07.860440 31415 leader_election.cc:302] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 821f593ade0a4a048ab73457258d39db; no voters:
I20260928 15:08:07.861114 31415 leader_election.cc:288] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:07.861236 31425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:07.863677 31415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:07.868383 31425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [term 1 LEADER]: Becoming Leader. State: Replica: 821f593ade0a4a048ab73457258d39db, State: Running, Role: LEADER
I20260928 15:08:07.869097 31425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f593ade0a4a048ab73457258d39db" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45481 } }
I20260928 15:08:07.872644 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "821f593ade0a4a048ab73457258d39db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f593ade0a4a048ab73457258d39db" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45481 } } }
I20260928 15:08:07.873011 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:07.873473 31427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 821f593ade0a4a048ab73457258d39db. Latest consensus state: current_term: 1 leader_uuid: "821f593ade0a4a048ab73457258d39db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f593ade0a4a048ab73457258d39db" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 45481 } } }
I20260928 15:08:07.873757 31427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:07.875809 31432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:07.876518 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:07.859754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38647 (local address 127.9.12.254:45481)
0928 15:08:07.859939 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:07.859946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:07.860810 (+ 864us) server_negotiation.cc:407] Connection header received
0928 15:08:07.861027 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:07.861036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:07.861114 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:07.861218 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:07.862899 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:07.863724 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:07.865035 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:07.865632 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:07.865765 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:07.868558 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:07.868566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:07.868570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:07.868610 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:07.871945 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:07.873083 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:07.873092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:07.873098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:07.873390 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:07.874048 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:07.874056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:07.874063 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:07.874462 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:07.874766 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:07.875983 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0928 15:08:07.876168 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:08:07.882385 31432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:07.893043 31432 catalog_manager.cc:1347] Generated new cluster ID: fd5354dc4e3c4bf484b5a33234aa279a
I20260928 15:08:07.893182 31432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:07.926306 31432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:07.928380 31432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:07.939438 31432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 821f593ade0a4a048ab73457258d39db: Generated new TSK 0
I20260928 15:08:07.940474 31432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:07.949841 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31348
I20260928 15:08:07.950979 31410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:08.042730 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31348
2026-09-28T15:08:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:08 dist-test-slave-dmhs krb5kdc[31447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:08 dist-test-slave-dmhs krb5kdc[31447](info): set up 2 sockets
Sep 28 15:08:08 dist-test-slave-dmhs krb5kdc[31447](info): commencing operation
krb5kdc: starting...
W20260928 15:08:09.145033 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:09 dist-test-slave-dmhs krb5kdc[31447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:09.399616 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40481
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:09.719424 31460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:09.719753 31460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:09.719902 31460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:09.726816 31460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:09.726994 31460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:09.727093 31460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:09.727216 31460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:09.727294 31460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:09.736634 31460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:09.737871 31460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:09.739130 31460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:09.750854 31465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:09.750806 31468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:09.750892 31466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:09.751770 31460 server_base.cc:1034] running on GCE node
I20260928 15:08:09.752652 31460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:09.754042 31460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:09.755270 31460 hybrid_clock.cc:648] HybridClock initialized: now 1790608089755232 us; error 47 us; skew 500 ppm
I20260928 15:08:09.755630 31460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:09 dist-test-slave-dmhs krb5kdc[31447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608089, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:09.758688 31460 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:09.760087 31460 webserver.cc:460] Webserver started at http://127.9.12.254:34123/ using document root <none> and password file <none>
I20260928 15:08:09.760815 31460 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:09.760967 31460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:09.761271 31460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:09.764233 31460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "f2cbb7f48cf540f196a31510fd00aa1f"
format_stamp: "Formatted at 2026-09-28 15:08:09 on dist-test-slave-dmhs"
server_key: "d2b4364705ab53dec61185052bf04a79"
server_key_iv: "04dbe17efeba1f97671d48132e83cd20"
server_key_version: "encryptionkey@0"
I20260928 15:08:09.765062 31460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "f2cbb7f48cf540f196a31510fd00aa1f"
format_stamp: "Formatted at 2026-09-28 15:08:09 on dist-test-slave-dmhs"
server_key: "d2b4364705ab53dec61185052bf04a79"
server_key_iv: "04dbe17efeba1f97671d48132e83cd20"
server_key_version: "encryptionkey@0"
I20260928 15:08:09.769991 31460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:08:09.773267 31475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:09.774493 31460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:08:09.774668 31460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "f2cbb7f48cf540f196a31510fd00aa1f"
format_stamp: "Formatted at 2026-09-28 15:08:09 on dist-test-slave-dmhs"
server_key: "d2b4364705ab53dec61185052bf04a79"
server_key_iv: "04dbe17efeba1f97671d48132e83cd20"
server_key_version: "encryptionkey@0"
I20260928 15:08:09.774847 31460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:09.789517 31460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:09.793388 31460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:09.793666 31460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:09.811626 31460 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:40481
I20260928 15:08:09.811664 31526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:40481 every 8 connection(s)
I20260928 15:08:09.813097 31460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:09.816601 31527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:09.822576 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31460
I20260928 15:08:09.822772 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:09.823233 9267 external_mini_cluster.cc:1428] Setting key f89e1c6d2f8179f4ec3baf2f01da6053
I20260928 15:08:09.827292 31527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f: Bootstrap starting.
I20260928 15:08:09.831355 31527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:09.832551 31527 log.cc:824] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:09.835680 31527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f: No bootstrap required, opened a new log
I20260928 15:08:09.838757 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:09.825068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38775 (local address 127.9.12.254:40481)
0928 15:08:09.825823 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:09.825838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:09.825885 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:08:09.828486 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:09.828513 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:09.829012 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:09.829475 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:09.829977 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:09.831983 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:09.833793 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:09.834756 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:09.834946 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:09.836047 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:09.836057 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:09.836073 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:09.836805 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:09.836895 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:09.836924 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:09.837059 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:09.837236 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 15:08:09.837499 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":298,"threads_started":1}
I20260928 15:08:09.842417 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:09.844743 31527 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40481 } }
I20260928 15:08:09.845156 31527 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:09.845265 31527 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cbb7f48cf540f196a31510fd00aa1f, State: Initialized, Role: FOLLOWER
I20260928 15:08:09.845956 31527 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40481 } }
I20260928 15:08:09.846186 31527 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:09.846330 31527 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:09.846501 31527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:09.849627 31527 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40481 } }
I20260928 15:08:09.850104 31527 leader_election.cc:302] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2cbb7f48cf540f196a31510fd00aa1f; no voters:
I20260928 15:08:09.850790 31527 leader_election.cc:288] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:09.850884 31537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:09.856400 31537 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [term 1 LEADER]: Becoming Leader. State: Replica: f2cbb7f48cf540f196a31510fd00aa1f, State: Running, Role: LEADER
I20260928 15:08:09.856986 31537 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40481 } }
I20260928 15:08:09.857345 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:09.846718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47331 (local address 127.9.12.254:40481)
0928 15:08:09.846938 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:09.846945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:09.847535 (+ 590us) server_negotiation.cc:407] Connection header received
0928 15:08:09.847656 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:09.847663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:09.847749 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:09.847845 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:09.848314 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:09.849392 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:09.850841 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:09.852084 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:09.852204 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:09.856560 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:09.856565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:09.856569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:09.856587 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:09.856626 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:09.856643 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:09.856751 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:09.856899 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 15:08:09.856990 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:08:09.857627 31527 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:09.860217 31538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40481 } } }
I20260928 15:08:09.860634 31538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:09.860426 31539 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2cbb7f48cf540f196a31510fd00aa1f. Latest consensus state: current_term: 1 leader_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cbb7f48cf540f196a31510fd00aa1f" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 40481 } } }
I20260928 15:08:09.860814 31539 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:09.862217 31542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:09.868424 31542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:09.876443 31542 catalog_manager.cc:1347] Generated new cluster ID: ff70f6c46d9c4aae9bc3d2a595b59a71
I20260928 15:08:09.876559 31542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:09.899303 31542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:09.900828 31542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:09.916365 31542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2cbb7f48cf540f196a31510fd00aa1f: Generated new TSK 0
I20260928 15:08:09.917402 31542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:09.925187 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31460
W20260928 15:08:09.926246 31522 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38775
W20260928 15:08:09.927031 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31460: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:08:09.927186 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31460
2026-09-28T15:08:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:10 dist-test-slave-dmhs krb5kdc[31558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:10 dist-test-slave-dmhs krb5kdc[31558](info): set up 2 sockets
Sep 28 15:08:10 dist-test-slave-dmhs krb5kdc[31558](info): commencing operation
krb5kdc: starting...
W20260928 15:08:11.024796 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:11 dist-test-slave-dmhs krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:11.277958 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37201
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:37201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:11.601459 31571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:11.601742 31571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:11.601837 31571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:11.609404 31571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:11.609508 31571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:11.609618 31571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:11.609707 31571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:11.609771 31571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:11.619422 31571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:37201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:37201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:11.620751 31571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:11.622059 31571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:11.633641 31579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:11.633724 31577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:11.633869 31576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:11.634284 31571 server_base.cc:1034] running on GCE node
I20260928 15:08:11.634974 31571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:11.636341 31571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:11.637557 31571 hybrid_clock.cc:648] HybridClock initialized: now 1790608091637517 us; error 32 us; skew 500 ppm
I20260928 15:08:11.637908 31571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:11 dist-test-slave-dmhs krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608091, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:11.644727 31571 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:11.646168 31571 webserver.cc:460] Webserver started at http://127.9.12.254:45415/ using document root <none> and password file <none>
I20260928 15:08:11.646852 31571 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:11.646986 31571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:11.647305 31571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:11.650590 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "6e926ddd77c14f00acb66a2ec432eb3b"
format_stamp: "Formatted at 2026-09-28 15:08:11 on dist-test-slave-dmhs"
server_key: "3524dd0ccaf04c9ecafd72b6c1c8721c"
server_key_iv: "5bdffbaf11979bf902f7819fdf0361fc"
server_key_version: "encryptionkey@0"
I20260928 15:08:11.651553 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "6e926ddd77c14f00acb66a2ec432eb3b"
format_stamp: "Formatted at 2026-09-28 15:08:11 on dist-test-slave-dmhs"
server_key: "3524dd0ccaf04c9ecafd72b6c1c8721c"
server_key_iv: "5bdffbaf11979bf902f7819fdf0361fc"
server_key_version: "encryptionkey@0"
I20260928 15:08:11.656605 31571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:08:11.659823 31586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:11.661096 31571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:08:11.661273 31571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "6e926ddd77c14f00acb66a2ec432eb3b"
format_stamp: "Formatted at 2026-09-28 15:08:11 on dist-test-slave-dmhs"
server_key: "3524dd0ccaf04c9ecafd72b6c1c8721c"
server_key_iv: "5bdffbaf11979bf902f7819fdf0361fc"
server_key_version: "encryptionkey@0"
I20260928 15:08:11.661468 31571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:11.666930 31571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:11.670759 31571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:11.671061 31571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:11.689109 31571 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:37201
I20260928 15:08:11.689149 31637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:37201 every 8 connection(s)
I20260928 15:08:11.690461 31571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:11.690989 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31571
I20260928 15:08:11.691179 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:11.691809 9267 external_mini_cluster.cc:1428] Setting key 1f0ef726e0da66b4e0d7589cebe25836
I20260928 15:08:11.695034 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:11.700594 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:11.693715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38807 (local address 127.9.12.254:37201)
0928 15:08:11.694523 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:11.694538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:11.694589 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:08:11.696917 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:11.696945 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:11.697357 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:11.697771 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:11.698750 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:11.698759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:11.698774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:11.698797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:11.698873 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:11.698897 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:11.699117 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:11.699273 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 15:08:11.699447 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":335,"threads_started":1}
I20260928 15:08:11.704224 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:11.708041 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b: Bootstrap starting.
I20260928 15:08:11.710975 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:11.708584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57489 (local address 127.9.12.254:37201)
0928 15:08:11.708746 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:11.708753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:11.709121 (+ 368us) server_negotiation.cc:407] Connection header received
0928 15:08:11.709294 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:11.709299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:11.709357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:11.709457 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:11.710344 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:11.710349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:11.710361 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:08:11.710380 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:11.710421 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:08:11.710435 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:11.710548 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:11.710670 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 15:08:11.710726 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:08:11.711583 31638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:11.712872 31638 log.cc:824] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:11.715492 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b: No bootstrap required, opened a new log
I20260928 15:08:11.722782 31638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37201 } }
I20260928 15:08:11.723090 31638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:08:11.723163 31638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e926ddd77c14f00acb66a2ec432eb3b, State: Initialized, Role: FOLLOWER
I20260928 15:08:11.723623 31638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37201 } }
I20260928 15:08:11.723812 31638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:08:11.723907 31638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:08:11.724021 31638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:11.726133 31638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37201 } }
I20260928 15:08:11.726557 31638 leader_election.cc:302] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e926ddd77c14f00acb66a2ec432eb3b; no voters:
I20260928 15:08:11.727137 31638 leader_election.cc:288] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:08:11.727283 31648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:11.728088 31648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [term 1 LEADER]: Becoming Leader. State: Replica: 6e926ddd77c14f00acb66a2ec432eb3b, State: Running, Role: LEADER
I20260928 15:08:11.728629 31648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37201 } }
I20260928 15:08:11.729326 31638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:11.731493 31649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37201 } } }
I20260928 15:08:11.732020 31649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:11.731638 31650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e926ddd77c14f00acb66a2ec432eb3b. Latest consensus state: current_term: 1 leader_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e926ddd77c14f00acb66a2ec432eb3b" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 37201 } } }
I20260928 15:08:11.732600 31650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:11.734159 31657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:11.739112 31657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:11.747766 31657 catalog_manager.cc:1347] Generated new cluster ID: b5285589976c4c6db3d3e7a1c3c72556
I20260928 15:08:11.747876 31657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:11.769820 31657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:11.770251 31657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:11.778061 31657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e926ddd77c14f00acb66a2ec432eb3b: Generated new TSK 0
I20260928 15:08:11.779018 31657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:11.791416 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31571
W20260928 15:08:11.792624 31633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38807
W20260928 15:08:11.793331 9267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31571: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:08:11.793493 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31571
2026-09-28T15:08:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1865 ms)
[----------] 18 tests from AuthTokenIssuingTest (34414 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:11 dist-test-slave-dmhs krb5kdc[31669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:11 dist-test-slave-dmhs krb5kdc[31669](info): set up 2 sockets
Sep 28 15:08:11 dist-test-slave-dmhs krb5kdc[31669](info): commencing operation
krb5kdc: starting...
W20260928 15:08:12.905277 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:13 dist-test-slave-dmhs krb5kdc[31669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:13.156289 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:39883
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:39883,127.9.12.253:44729,127.9.12.252:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:13.480557 31682 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:08:13.480897 31682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:13.481012 31682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:13.481142 31682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:13.488133 31682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:13.488245 31682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:13.488358 31682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:13.488449 31682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:13.488576 31682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:13.498221 31682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:39883,127.9.12.253:44729,127.9.12.252:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:13.499472 31682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:13.500821 31682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:13.512341 31690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:13.512408 31688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:13.512430 31687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:13.513129 31682 server_base.cc:1034] running on GCE node
I20260928 15:08:13.513887 31682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:13.515156 31682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:13.516383 31682 hybrid_clock.cc:648] HybridClock initialized: now 1790608093516314 us; error 37 us; skew 500 ppm
I20260928 15:08:13.516729 31682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:13 dist-test-slave-dmhs krb5kdc[31669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:13.519716 31682 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:13.521201 31682 webserver.cc:460] Webserver started at http://127.9.12.254:34537/ using document root <none> and password file <none>
I20260928 15:08:13.521864 31682 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:13.521992 31682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:13.522326 31682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:13.525219 31682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "204239ef99e44b808f20069558a3d560"
format_stamp: "Formatted at 2026-09-28 15:08:13 on dist-test-slave-dmhs"
server_key: "b2710e0d9647e2f7ba9291b424a76d47"
server_key_iv: "cdb13cd224cd80980ba01cb36d2bbdcd"
server_key_version: "encryptionkey@0"
I20260928 15:08:13.525972 31682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "204239ef99e44b808f20069558a3d560"
format_stamp: "Formatted at 2026-09-28 15:08:13 on dist-test-slave-dmhs"
server_key: "b2710e0d9647e2f7ba9291b424a76d47"
server_key_iv: "cdb13cd224cd80980ba01cb36d2bbdcd"
server_key_version: "encryptionkey@0"
I20260928 15:08:13.530844 31682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:08:13.534070 31697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:13.535347 31682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:08:13.535537 31682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "204239ef99e44b808f20069558a3d560"
format_stamp: "Formatted at 2026-09-28 15:08:13 on dist-test-slave-dmhs"
server_key: "b2710e0d9647e2f7ba9291b424a76d47"
server_key_iv: "cdb13cd224cd80980ba01cb36d2bbdcd"
server_key_version: "encryptionkey@0"
I20260928 15:08:13.535748 31682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:13.552927 31682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:13.556753 31682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:13.557031 31682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:13.575615 31682 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:39883
I20260928 15:08:13.575645 31748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:39883 every 8 connection(s)
I20260928 15:08:13.576962 31682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:13.579893 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31682
I20260928 15:08:13.580109 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:13.580575 31749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:13.580839 9267 external_mini_cluster.cc:1428] Setting key 985b2427bc6dc8dd90b8bb9e0e8d476d
I20260928 15:08:13.586863 31749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:13 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.253@KRBTEST.COM" created.
I20260928 15:08:13.611353 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.589901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39883 (local address 127.0.0.1:43824)
0928 15:08:13.591054 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.591097 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:13.593797 (+ 2700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:13.594605 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:13.594616 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:13.595408 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:13.595907 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:13.595931 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:13.597557 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:13.597565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:13.599393 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:13.599410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:13.600404 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:13.600414 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:13.600654 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:13.601326 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:13.601359 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:13.603988 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:13.607270 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:13.607281 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:13.607293 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:13.607664 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:13.608060 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:13.608066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:13.608070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:13.608296 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:13.609071 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:13.609081 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:13.609279 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 15:08:13.609513 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 15:08:13.610073 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":149,"spinlock_wait_cycles":34688,"thread_start_us":675,"threads_started":1}
I20260928 15:08:13.611371 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.589902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.9.12.254:39883)
0928 15:08:13.590770 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:13.590790 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:13.591222 (+ 432us) server_negotiation.cc:407] Connection header received
0928 15:08:13.594146 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:13.594152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:13.594368 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:13.594536 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:13.596109 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:13.597415 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:13.599607 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:13.600125 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:13.600301 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:13.604204 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:13.604212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:13.604223 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:13.604253 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:13.607136 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:13.607791 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:13.607805 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:13.607808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:13.607948 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:13.608403 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:13.608410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:13.608418 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:13.608920 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:13.609131 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:13.609634 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 15:08:13.609961 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":602,"thread_start_us":353,"threads_started":1}
I20260928 15:08:13.614434 31749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:13.615628 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.615163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59858)
0928 15:08:13.615354 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.615481 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 15:08:13.615916 31699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
I20260928 15:08:13.617448 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.617091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59874)
0928 15:08:13.617254 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.617316 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:08:13.617676 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:44729: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab.
Entry for principal kudu/127.9.12.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab.
I20260928 15:08:13.637813 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:44729
--webserver_interface=127.9.12.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab
--principal=kudu/127.9.12.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:39883,127.9.12.253:44729,127.9.12.252:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:08:13.666999 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } attempt: 1
I20260928 15:08:13.668255 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.667692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59884)
0928 15:08:13.667919 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.668072 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:08:13.669345 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.668931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59890)
0928 15:08:13.669083 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.669161 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:08:13.669529 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:44729: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
I20260928 15:08:13.737784 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } attempt: 2
I20260928 15:08:13.739023 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.738536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59906)
0928 15:08:13.738763 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.738879 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 15:08:13.740144 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.739753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59908)
0928 15:08:13.739921 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.740017 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 15:08:13.740418 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:44729: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
I20260928 15:08:13.831661 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } attempt: 3
I20260928 15:08:13.832839 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.832434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59912)
0928 15:08:13.832618 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.832704 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 15:08:13.833998 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.833644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59922)
0928 15:08:13.833802 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.833868 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 15:08:13.834244 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:44729: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
W20260928 15:08:13.961992 31756 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:08:13.962345 31756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:13.962471 31756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:13.962610 31756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:13.969729 31756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:13.969817 31756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:13.969923 31756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:13.970017 31756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:13.970105 31756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:13.977516 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } attempt: 4
I20260928 15:08:13.978858 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.978234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59932)
0928 15:08:13.978558 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.978647 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260928 15:08:13.979888 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:13.979538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59940)
0928 15:08:13.979685 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:13.979761 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:08:13.980144 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:44729: Network error: Client connection negotiation failed: client connection to 127.9.12.253:44729: connect: Connection refused (error 111)
I20260928 15:08:13.980674 31756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:39883,127.9.12.253:44729,127.9.12.252:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab
--principal=kudu/127.9.12.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:44729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.12.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:13.981971 31756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:13.983297 31756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:13.994872 31761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:13.995093 31762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:13.995124 31764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:13.995934 31756 server_base.cc:1034] running on GCE node
I20260928 15:08:13.996695 31756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:13.998083 31756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:13.999313 31756 hybrid_clock.cc:648] HybridClock initialized: now 1790608093999233 us; error 54 us; skew 500 ppm
I20260928 15:08:13.999635 31756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:13 dist-test-slave-dmhs krb5kdc[31669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:14.002696 31756 init.cc:377] Logged in from keytab as kudu/127.9.12.253@KRBTEST.COM (short username kudu)
I20260928 15:08:14.004120 31756 webserver.cc:460] Webserver started at http://127.9.12.253:37559/ using document root <none> and password file <none>
I20260928 15:08:14.004805 31756 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:14.004890 31756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:14.005185 31756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:14.008045 31756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data/instance:
uuid: "29c6526fa1da4be98343530441309661"
format_stamp: "Formatted at 2026-09-28 15:08:14 on dist-test-slave-dmhs"
server_key: "516934967e2b39c73658f0b8d9e78f44"
server_key_iv: "ea98e5caf9b2e29aa62d05272111d8ba"
server_key_version: "encryptionkey@0"
I20260928 15:08:14.008908 31756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal/instance:
uuid: "29c6526fa1da4be98343530441309661"
format_stamp: "Formatted at 2026-09-28 15:08:14 on dist-test-slave-dmhs"
server_key: "516934967e2b39c73658f0b8d9e78f44"
server_key_iv: "ea98e5caf9b2e29aa62d05272111d8ba"
server_key_version: "encryptionkey@0"
I20260928 15:08:14.013928 31756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:08:14.017136 31771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:14.018458 31756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:08:14.018642 31756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal
uuid: "29c6526fa1da4be98343530441309661"
format_stamp: "Formatted at 2026-09-28 15:08:14 on dist-test-slave-dmhs"
server_key: "516934967e2b39c73658f0b8d9e78f44"
server_key_iv: "ea98e5caf9b2e29aa62d05272111d8ba"
server_key_version: "encryptionkey@0"
I20260928 15:08:14.018853 31756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:14.034686 31756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:14.038422 31756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:14.038689 31756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:14.056787 31756 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.253:44729
I20260928 15:08:14.056828 31822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.253:44729 every 8 connection(s)
I20260928 15:08:14.058099 31756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
I20260928 15:08:14.060343 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31756
I20260928 15:08:14.060555 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-1/wal/instance
I20260928 15:08:14.060978 9267 external_mini_cluster.cc:1428] Setting key 7b431ebc540113ed1c72da92f3cda56e
I20260928 15:08:14.061828 31823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:14.068038 31823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.252@KRBTEST.COM" created.
I20260928 15:08:14.089372 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.070738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.9.12.254:39883)
0928 15:08:14.070949 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.070956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.072182 (+ 1226us) server_negotiation.cc:407] Connection header received
0928 15:08:14.074070 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.074078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.074176 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.074299 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.075737 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.076925 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.078408 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.079177 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.079319 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.082931 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.082939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.082946 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.082988 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.086253 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.086938 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.086941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.086944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.087104 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.087493 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.087496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.087498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.087941 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.088094 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.088983 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.089151 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:08:14.090938 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.071199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39883 (local address 127.0.0.1:43834)
0928 15:08:14.071981 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.072032 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.073801 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.074505 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.074513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.075010 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.075559 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.075576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.077088 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.077092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.078251 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.078267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.079378 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.079382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.079605 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.080324 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.080347 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.082694 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.086378 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.086384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.086395 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.086800 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.087226 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.087229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.087231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.087400 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.088096 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.088103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.088594 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.088900 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.089462 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":356,"threads_started":1}
I20260928 15:08:14.092638 31823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:08:14.108191 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.093604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.9.12.253:44729)
0928 15:08:14.094338 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.094348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.094380 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:08:14.094438 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.094448 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.094685 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.094836 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.095501 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.097170 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.098774 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.099310 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.099401 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.102869 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.102879 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.102890 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.102916 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.105526 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.106107 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.106121 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.106123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.106232 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.106570 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.106578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.106580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.106913 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.107167 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.107586 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.107894 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":540,"thread_start_us":404,"threads_started":1}
I20260928 15:08:14.108296 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.093497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59948)
0928 15:08:14.093717 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.093736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.093845 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.094838 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.094841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.095191 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.095333 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.095340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.097307 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.097310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.098633 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.098651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.099455 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.099464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.099685 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.100240 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.100281 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.102667 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.105647 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.105654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.105659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.105978 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.106334 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.106340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.106343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.106472 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.107121 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.107129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.107293 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.107460 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.107780 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:08:14.110531 31823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:14.111503 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.111128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60466)
0928 15:08:14.111280 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.111386 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:08:14.111815 31773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.113319 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.113041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60468)
0928 15:08:14.113166 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.113232 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:08:14.113631 31823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab.
Entry for principal kudu/127.9.12.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab.
I20260928 15:08:14.121856 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:34449
--webserver_interface=127.9.12.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab
--principal=kudu/127.9.12.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:39883,127.9.12.253:44729,127.9.12.252:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:08:14.162961 31823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 1
I20260928 15:08:14.164165 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.163724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60470)
0928 15:08:14.163912 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.164020 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 15:08:14.165176 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.164864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60482)
0928 15:08:14.165002 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.165082 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:08:14.165406 31823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.233645 31823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 2
I20260928 15:08:14.234805 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.234346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60496)
0928 15:08:14.234551 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.234639 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:08:14.235783 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.235500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60510)
0928 15:08:14.235623 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.235682 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:08:14.236009 31823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.279452 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } attempt: 5
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:08:14.294214 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.280402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.9.12.253:44729)
0928 15:08:14.280592 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.280598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.280620 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:08:14.280857 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.280862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.280959 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.281063 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.282035 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.282720 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.284436 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.284886 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.284981 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.288689 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.288694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.288697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.288727 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.291382 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.292306 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.292314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.292319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.292450 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.292903 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.292907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.292909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.293176 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.293284 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.293831 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.293995 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:08:14.294371 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.280285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:59956)
0928 15:08:14.280522 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.280543 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.280686 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.281140 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.281145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.281655 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.281884 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.281897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.282854 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.282860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.284255 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.284307 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.285035 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.285041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.285254 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.285894 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.285918 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.288493 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.291532 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.291539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.291545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.292125 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.292575 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.292583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.292587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.292785 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.293306 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.293312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.293534 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.293729 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.294026 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:08:14.295409 31749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:14.296404 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.295987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60516)
0928 15:08:14.296133 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.296203 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:08:14.297297 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.297019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60520)
0928 15:08:14.297135 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.297209 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 15:08:14.297549 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.327222 31823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 3
I20260928 15:08:14.328504 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.327939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60524)
0928 15:08:14.328187 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.328339 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:08:14.329677 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.329173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60534)
0928 15:08:14.329453 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.329541 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260928 15:08:14.329895 31823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.348769 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 1
I20260928 15:08:14.349857 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.349427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60536)
0928 15:08:14.349602 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.349686 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 15:08:14.350847 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.350568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:60544)
0928 15:08:14.350691 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.350752 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 15:08:14.351118 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.419354 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 2
I20260928 15:08:14.420576 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.420064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55560)
0928 15:08:14.420338 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.420423 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:08:14.421597 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.421270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55574)
0928 15:08:14.421390 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.421487 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:08:14.421849 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
W20260928 15:08:14.449311 31830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:08:14.449637 31830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:14.449750 31830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:14.449903 31830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:14.458914 31830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:14.459055 31830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:14.459152 31830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:14.459246 31830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:14.459357 31830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:14.469061 31830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:39883,127.9.12.253:44729,127.9.12.252:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab
--principal=kudu/127.9.12.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.12.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:14.470319 31830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:14.471616 31830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:08:14.473120 31823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 4
I20260928 15:08:14.474274 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.473748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55580)
0928 15:08:14.473989 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.474089 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:08:14.475451 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.475084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55594)
0928 15:08:14.475239 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.475316 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:08:14.475670 31823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
W20260928 15:08:14.483702 31835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:14.483740 31836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:14.483925 31838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:14.484589 31830 server_base.cc:1034] running on GCE node
I20260928 15:08:14.485340 31830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:14.486910 31830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:14.488112 31830 hybrid_clock.cc:648] HybridClock initialized: now 1790608094488043 us; error 45 us; skew 500 ppm
I20260928 15:08:14.488451 31830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608094, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:14.491549 31830 init.cc:377] Logged in from keytab as kudu/127.9.12.252@KRBTEST.COM (short username kudu)
I20260928 15:08:14.492947 31830 webserver.cc:460] Webserver started at http://127.9.12.252:40309/ using document root <none> and password file <none>
I20260928 15:08:14.493589 31830 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:14.493673 31830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:14.493927 31830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:14.496841 31830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data/instance:
uuid: "db7008afa7464e3393d26ff661d6b444"
format_stamp: "Formatted at 2026-09-28 15:08:14 on dist-test-slave-dmhs"
server_key: "7e24952bb66db5a33932a8dc065a0bfb"
server_key_iv: "0b06886009ed9cfaa1705c27a50fc31a"
server_key_version: "encryptionkey@0"
I20260928 15:08:14.497550 31830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal/instance:
uuid: "db7008afa7464e3393d26ff661d6b444"
format_stamp: "Formatted at 2026-09-28 15:08:14 on dist-test-slave-dmhs"
server_key: "7e24952bb66db5a33932a8dc065a0bfb"
server_key_iv: "0b06886009ed9cfaa1705c27a50fc31a"
server_key_version: "encryptionkey@0"
I20260928 15:08:14.502435 31830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 15:08:14.505759 31845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:14.507082 31830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:08:14.507244 31830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal
uuid: "db7008afa7464e3393d26ff661d6b444"
format_stamp: "Formatted at 2026-09-28 15:08:14 on dist-test-slave-dmhs"
server_key: "7e24952bb66db5a33932a8dc065a0bfb"
server_key_iv: "0b06886009ed9cfaa1705c27a50fc31a"
server_key_version: "encryptionkey@0"
I20260928 15:08:14.507436 31830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:14.522996 31830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:14.527316 31830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:14.527602 31830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:14.528075 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 3
I20260928 15:08:14.529307 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.528736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55598)
0928 15:08:14.528939 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.529064 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:08:14.530405 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.529964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55610)
0928 15:08:14.530164 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.530264 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 15:08:14.530612 31749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:34449: Network error: Client connection negotiation failed: client connection to 127.9.12.252:34449: connect: Connection refused (error 111)
I20260928 15:08:14.546458 31830 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.252:34449
I20260928 15:08:14.546491 31896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.252:34449 every 8 connection(s)
I20260928 15:08:14.547773 31830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
I20260928 15:08:14.551318 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:14.555608 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 31830
I20260928 15:08:14.555796 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607741729566-9267-0/minicluster-data/master-2/wal/instance
I20260928 15:08:14.556331 9267 external_mini_cluster.cc:1428] Setting key 540ebf019c479f89131882f62c7021d1
I20260928 15:08:14.558598 31897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608094, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:08:14.577937 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.561092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.9.12.254:39883)
0928 15:08:14.561263 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.561267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.561912 (+ 645us) server_negotiation.cc:407] Connection header received
0928 15:08:14.562145 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.562153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.562255 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.562332 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.564144 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.564888 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.567045 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.567525 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.567657 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.571164 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.571169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.571172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.571205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.574380 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.575332 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.575337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.575340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.575513 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.575969 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.575974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.575977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.576407 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.576559 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.577344 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.577557 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:08:14.579071 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.561114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:39883 (local address 127.0.0.1:50344)
0928 15:08:14.561774 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.561809 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.561986 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.562605 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.562613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.563439 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.563964 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.563983 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.565041 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.565046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.566890 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.566906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.567760 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.567766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.567874 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.568644 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.568665 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.570973 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.574516 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.574525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.574536 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.575143 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.575624 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.575630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.575635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.575847 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.576546 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.576555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.576959 (+ 404us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.577254 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.577715 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":343,"threads_started":1}
I20260928 15:08:14.579427 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.558179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.9.12.252:34449)
0928 15:08:14.558956 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.558967 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.559021 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:08:14.561010 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.561044 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.561601 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.561981 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.563883 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.565142 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.566914 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.567400 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.567533 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.572196 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.572206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.572240 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.572309 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.575439 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.576266 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.576294 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.576298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.576427 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.576858 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.576863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.576865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.577269 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.577521 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.578763 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.578976 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":567,"thread_start_us":368,"threads_started":1}
I20260928 15:08:14.580789 31897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608094, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:08:14.594928 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.581470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:44729 (local address 127.0.0.1:54384)
0928 15:08:14.581639 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.581657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.581751 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.582763 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.582768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.583206 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.583351 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.583360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.584362 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.584369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.585817 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.585833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.586534 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.586540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.586743 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.587269 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.587292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.589406 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.592641 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.592646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.592650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.592996 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.593378 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.593382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.593386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.593534 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.594017 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.594021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.594170 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.594309 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.594576 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:08:14.594967 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.582145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.9.12.253:44729)
0928 15:08:14.582311 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.582316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.582335 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:08:14.582396 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.582401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.582476 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.582591 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.583502 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.584191 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.585943 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.586377 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.586512 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.589597 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.589602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.589605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.589637 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.592505 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.593120 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.593131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.593134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.593258 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.593642 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.593648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.593652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.593900 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.594060 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.594398 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.594607 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:08:14.595834 31897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:14.596796 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.582482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.9.12.254:39883)
0928 15:08:14.582634 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.582638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.582755 (+ 117us) server_negotiation.cc:407] Connection header received
0928 15:08:14.582864 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.582869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.582947 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.583060 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.584531 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.585716 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.586907 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.587513 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.587641 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.591473 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.591478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.591482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.591519 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.594036 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.594664 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.594687 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.594691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.594801 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.595256 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.595261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.595264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.595585 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.595862 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.596159 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.596420 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608094, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:08:14.608577 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.596693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.9.12.252:34449)
0928 15:08:14.596873 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.596880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.597113 (+ 233us) server_negotiation.cc:407] Connection header received
0928 15:08:14.597277 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.597282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.597342 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.597433 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.598233 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.599036 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.600155 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.600595 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.600719 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.603365 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.603369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.603372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.603398 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.605901 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.606629 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.606633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.606635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.606734 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.607171 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.607176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.607179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.607519 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.607670 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.607968 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.608195 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":67}
I20260928 15:08:14.608668 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.596493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55624)
0928 15:08:14.597035 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.597048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.597161 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.597446 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.597450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.597906 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.598064 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.598074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.599164 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.599170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.600028 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.600039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.600724 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.600729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.600971 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.601442 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.601460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.603190 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.606034 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.606038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.606040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.606516 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.606877 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.606885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.606889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.607065 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.607635 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.607638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.607757 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.607896 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.608320 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":152}
I20260928 15:08:14.613225 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.598595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.9.12.253:44729)
0928 15:08:14.598724 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.598728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.598740 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:08:14.598773 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.598776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.598821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.598922 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.600087 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.600832 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.602119 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.602553 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.602683 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.607163 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.607169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.607173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.607211 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.610509 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.611238 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.611243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.611246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.611422 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.611773 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.611779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.611782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.612134 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.612429 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.612675 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.612890 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:08:14.614492 31897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: Bootstrap starting.
I20260928 15:08:14.617825 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:14.618119 31897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:14.619169 31897 log.cc:824] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:14.621730 31897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: No bootstrap required, opened a new log
I20260928 15:08:14.620453 31724 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "204239ef99e44b808f20069558a3d560"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50348
I20260928 15:08:14.628870 31897 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.629202 31897 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7008afa7464e3393d26ff661d6b444, State: Initialized, Role: FOLLOWER
I20260928 15:08:14.629621 31897 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.630537 31903 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.630914 31903 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:14.631548 31897 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:08:14.641181 31914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:08:14.641314 31914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:08:14.638527 31830 master_runner.cc:186] Error getting master registration for 127.9.12.254:39883: OK, instance_id { permanent_uuid: "204239ef99e44b808f20069558a3d560" instance_seqno: 1790608093570396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:08:14.649727 31830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:08:14.707846 31749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 4
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:08:14.724807 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.708697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55632)
0928 15:08:14.708899 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.708922 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.709098 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.709488 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.709494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.710002 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.710200 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.710210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.711664 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.711669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.713197 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.713212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.714225 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.714231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.714442 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.715170 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.715197 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.718057 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.722192 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.722200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.722205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.722611 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.723017 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.723022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.723026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.723215 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.723977 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.723980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.724136 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.724247 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.724545 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:08:14.725160 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.708878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.9.12.252:34449)
0928 15:08:14.709096 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.709103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.709125 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:08:14.709229 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.709239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.709327 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.709452 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.710342 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.711510 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.713351 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.714071 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.714198 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.718258 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.718265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.718270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.718311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.722067 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.722734 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.722741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.722745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.722887 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.723331 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.723337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.723341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.723797 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.724012 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.724452 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.724728 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:08:14.729656 31749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: Bootstrap starting.
I20260928 15:08:14.733224 31749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:14.734490 31749 log.cc:824] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:14.741595 31749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: No bootstrap required, opened a new log
I20260928 15:08:14.749260 31749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.749692 31749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204239ef99e44b808f20069558a3d560, State: Initialized, Role: FOLLOWER
I20260928 15:08:14.750190 31749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.751444 31917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.752079 31917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:14.752722 31749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:14.752766 31724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:08:14.752954 31724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:14.756954 31724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.760257 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "204239ef99e44b808f20069558a3d560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "db7008afa7464e3393d26ff661d6b444"
I20260928 15:08:14.759260 31724 leader_election.cc:288] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [CANDIDATE]: Term 1 election: Requested vote from peers 29c6526fa1da4be98343530441309661 (127.9.12.253:44729), db7008afa7464e3393d26ff661d6b444 (127.9.12.252:34449)
I20260928 15:08:14.760831 31872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:14.763984 31872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 204239ef99e44b808f20069558a3d560 in term 1.
I20260928 15:08:14.764648 31699 leader_election.cc:302] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204239ef99e44b808f20069558a3d560, db7008afa7464e3393d26ff661d6b444; no voters:
I20260928 15:08:14.765311 31917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:14.770416 31917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [term 1 LEADER]: Becoming Leader. State: Replica: 204239ef99e44b808f20069558a3d560, State: Running, Role: LEADER
I20260928 15:08:14.771211 31917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.774899 31823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } attempt: 5
I20260928 15:08:14.763912 31682 master_runner.cc:186] Error getting master registration for 127.9.12.253:44729: OK, instance_id { permanent_uuid: "29c6526fa1da4be98343530441309661" instance_seqno: 1790608094051679 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:08:14.776909 31682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 15:08:14.777328 31928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:08:14.777509 31928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 15:08:14.779960 31851 tablet.cc:2369] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:08:14 dist-test-slave-dmhs krb5kdc[31669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:08:14.787621 31931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204239ef99e44b808f20069558a3d560. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.788372 31931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:14.795445 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.775489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:34449 (local address 127.0.0.1:55636)
0928 15:08:14.775855 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:14.775877 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:14.775995 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:14.776999 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:14.777006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:14.777601 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:14.777836 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.777852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.779319 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.779328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.781477 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:14.781498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.782740 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.782749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:14.783106 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.783993 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:14.784029 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:14.787402 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:14.792538 (+ 5136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.792544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.792548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.793085 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.793558 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:14.793564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:14.793567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:14.793729 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:14.794428 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:14.794434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:14.794591 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 15:08:14.794744 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:08:14.795048 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260928 15:08:14.796742 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.776219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.9.12.252:34449)
0928 15:08:14.776498 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.776505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.776524 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:08:14.776584 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.776589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.776691 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.776798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.778010 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.779121 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.781655 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.782543 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.782710 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.787692 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.787702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.787709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.787765 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.792366 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.793230 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.793236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.793240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.793416 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.793874 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.793879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.793883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.794211 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.794365 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.795925 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.796226 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 15:08:14.798926 31935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:14.802871 31823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: Bootstrap starting.
I20260928 15:08:14.803126 31935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:14.807734 31823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:14.809312 31823 log.cc:824] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: Log is configured to *not* fsync() on all Append() calls
W20260928 15:08:14.809660 31703 tablet.cc:2369] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:08:14.826402 31872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [term 1 FOLLOWER]: Refusing update from remote peer 204239ef99e44b808f20069558a3d560: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:08:14.827617 31931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:08:14.839551 31823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: No bootstrap required, opened a new log
I20260928 15:08:14.856024 31903 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204239ef99e44b808f20069558a3d560. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.856376 31903 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:14.875536 31903 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.875833 31903 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:14.877106 31917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.877434 31917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:14.879699 31823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.880193 31823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c6526fa1da4be98343530441309661, State: Initialized, Role: FOLLOWER
I20260928 15:08:14.880810 31823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } }
I20260928 15:08:14.898607 31935 catalog_manager.cc:1347] Generated new cluster ID: 1d02a720df4841aa875d39607199c201
I20260928 15:08:14.898729 31935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:14.899147 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.819460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.9.12.254:39883)
0928 15:08:14.819820 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.819825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.842752 (+ 22927us) server_negotiation.cc:407] Connection header received
0928 15:08:14.842956 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.842961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.843049 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.843145 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.847824 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.849097 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.858434 (+ 9337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.859136 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.859248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.877633 (+ 18385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.877641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.877646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.877695 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.881720 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.892235 (+ 10515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.892244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.892252 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.892562 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.893678 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.893686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.893692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.894218 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.894520 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.896756 (+ 2236us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.897914 (+ 1158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260928 15:08:14.913884 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.819634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.9.12.253:44729)
0928 15:08:14.819819 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.819825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.845252 (+ 25427us) server_negotiation.cc:407] Connection header received
0928 15:08:14.845407 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.845412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.845481 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.845588 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.859772 (+ 14184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.860878 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.880300 (+ 19422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.881027 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.881136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.887269 (+ 6133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.887275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.887281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.887327 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.891952 (+ 4625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.907112 (+ 15160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.907121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.907128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.907357 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.909409 (+ 2052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.909417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.909423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.909942 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.910237 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.912999 (+ 2762us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.913343 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:08:14.922809 31946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.923449 31946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:14.924494 31823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:14.924932 31797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:14.929767 31797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [term 1 FOLLOWER]: Refusing update from remote peer 204239ef99e44b808f20069558a3d560: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:08:14.930924 31798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "204239ef99e44b808f20069558a3d560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29c6526fa1da4be98343530441309661"
I20260928 15:08:14.935549 31798 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 204239ef99e44b808f20069558a3d560 in term 1.
I20260928 15:08:14.936666 31931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:08:14.942730 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:14.843419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.9.12.252:34449)
0928 15:08:14.843600 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:14.843608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:14.853529 (+ 9921us) server_negotiation.cc:407] Connection header received
0928 15:08:14.854657 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:14.854664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:14.854778 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:14.854865 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:14.871075 (+ 16210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.872223 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.883583 (+ 11360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:14.884323 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:14.884432 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:14.918060 (+ 33628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:14.918071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:14.918079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:14.918139 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:14.937820 (+ 19681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.939367 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.939373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.939379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.939586 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:14.940225 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:14.940233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:14.940238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:14.940787 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:14.941101 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:14.941842 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 15:08:14.942203 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:08:14.950910 31935 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:14.953971 31935 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:14.989758 31935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560: Generated new TSK 0
I20260928 15:08:14.990865 31935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:08:14.997478 31946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204239ef99e44b808f20069558a3d560. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.997929 31946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:14.998405 31951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:14.998827 31951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:15.003101 31931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "204239ef99e44b808f20069558a3d560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204239ef99e44b808f20069558a3d560" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 39883 } } peers { permanent_uuid: "29c6526fa1da4be98343530441309661" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 44729 } } peers { permanent_uuid: "db7008afa7464e3393d26ff661d6b444" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 34449 } } }
I20260928 15:08:15.003386 31931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204239ef99e44b808f20069558a3d560 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:15.074048 31963 catalog_manager.cc:1259] Loaded cluster ID: 1d02a720df4841aa875d39607199c201
I20260928 15:08:15.074162 31963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: loading cluster ID for follower catalog manager: success
I20260928 15:08:15.077808 31963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: acquiring CA information for follower catalog manager: success
I20260928 15:08:15.080214 31963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29c6526fa1da4be98343530441309661: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:08:15.110137 9267 mini_kdc.cc:348] Time spent kdestroy: real 0.103s user 0.003s sys 0.023s
W20260928 15:08:15.165896 31752 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.143901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.9.12.254:39883)
0928 15:08:15.144107 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.144113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.161699 (+ 17586us) server_negotiation.cc:407] Connection header received
0928 15:08:15.164276 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.164283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.164381 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.164466 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.165629 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50364: BlockingRecv error: recv got EOF from 127.0.0.1:50364 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 15:08:15.169952 31827 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.146054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.9.12.253:44729)
0928 15:08:15.146270 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.146277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.167677 (+ 21400us) server_negotiation.cc:407] Connection header received
0928 15:08:15.168549 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.168554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.168633 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.168701 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.169722 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54398: BlockingRecv error: recv got EOF from 127.0.0.1:54398 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 15:08:15.171144 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.162268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.9.12.252:34449)
0928 15:08:15.162448 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.162453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.168168 (+ 5715us) server_negotiation.cc:407] Connection header received
0928 15:08:15.168364 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.168371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.168461 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.168557 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.170909 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55654: BlockingRecv error: recv got EOF from 127.0.0.1:55654 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:08:15.229533 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.210044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.9.12.253:44729)
0928 15:08:15.210236 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.210242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.220918 (+ 10676us) server_negotiation.cc:407] Connection header received
0928 15:08:15.221455 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.221462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.221697 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.221891 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.222700 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.223891 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.226464 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.227188 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.227290 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.228033 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.228712 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.228840 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.229083 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.229178 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:08:15.248440 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.245968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.9.12.252:34449)
0928 15:08:15.246174 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.246180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.246722 (+ 542us) server_negotiation.cc:407] Connection header received
0928 15:08:15.246986 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.247001 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.247139 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.247266 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.248130 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55670: BlockingRecv error: recv got EOF from 127.0.0.1:55670 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:08:15.259492 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.253008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.9.12.253:44729)
0928 15:08:15.253186 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.253191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.253214 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:08:15.253303 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.253308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.253397 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.253518 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.254101 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.255267 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.257329 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.258049 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.258189 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.258656 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.258803 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.258897 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.259056 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.259143 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 15:08:15.270277 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.266926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.9.12.252:34449)
0928 15:08:15.267154 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.267161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.268781 (+ 1620us) server_negotiation.cc:407] Connection header received
0928 15:08:15.268938 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.268944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.269036 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.269140 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.270049 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55684: BlockingRecv error: recv got EOF from 127.0.0.1:55684 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:08:15.286403 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.275925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.9.12.253:44729)
0928 15:08:15.276097 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.276102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.280307 (+ 4205us) server_negotiation.cc:407] Connection header received
0928 15:08:15.280510 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.280516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.280613 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.280707 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.281183 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.282303 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.284327 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.284979 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.285103 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.285559 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.285704 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.285790 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.285968 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.286070 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 15:08:15.302330 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.298320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.9.12.252:34449)
0928 15:08:15.298502 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.298508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.300852 (+ 2344us) server_negotiation.cc:407] Connection header received
0928 15:08:15.301045 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.301052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.301145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.301250 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.302067 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55690: BlockingRecv error: recv got EOF from 127.0.0.1:55690 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:08:15.321230 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.314417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.9.12.253:44729)
0928 15:08:15.314657 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.314663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.314685 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:08:15.314945 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.314952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.315042 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.315104 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.315717 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.316939 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.318979 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.319728 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.319866 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.320454 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.320592 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.320696 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.320854 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.320945 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260928 15:08:15.330214 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.328512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.9.12.252:34449)
0928 15:08:15.328780 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.328786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.328806 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:08:15.328875 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.328880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.328987 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.329142 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.329983 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55692: BlockingRecv error: recv got EOF from 127.0.0.1:55692 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260928 15:08:15.352960 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.345769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.9.12.253:44729)
0928 15:08:15.345974 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.345982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.346495 (+ 513us) server_negotiation.cc:407] Connection header received
0928 15:08:15.346691 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.346699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.346818 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.346939 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.347429 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.348666 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.350626 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.351343 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.351497 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.352010 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.352186 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.352333 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.352538 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.352633 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 15:08:15.368220 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.366045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.9.12.252:34449)
0928 15:08:15.366217 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.366225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.366758 (+ 533us) server_negotiation.cc:407] Connection header received
0928 15:08:15.366965 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.366972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.367082 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.367200 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.368014 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55706: BlockingRecv error: recv got EOF from 127.0.0.1:55706 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:08:15.396289 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.388718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.9.12.253:44729)
0928 15:08:15.388911 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.388917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.389459 (+ 542us) server_negotiation.cc:407] Connection header received
0928 15:08:15.389668 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.389674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.389766 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.389869 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.390364 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.391669 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.394156 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.394807 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.394937 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.395451 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.395598 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.395693 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.395865 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.395955 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:08:15.405463 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.403159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.9.12.252:34449)
0928 15:08:15.403377 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.403386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.403987 (+ 601us) server_negotiation.cc:407] Connection header received
0928 15:08:15.404151 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.404159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.404282 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.404386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.405193 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55722: BlockingRecv error: recv got EOF from 127.0.0.1:55722 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:08:15.447466 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.440948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.9.12.253:44729)
0928 15:08:15.441187 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.441196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.441592 (+ 396us) server_negotiation.cc:407] Connection header received
0928 15:08:15.441771 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.441777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.441913 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.442047 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.442442 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.443589 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.444939 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.445697 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.445841 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.446408 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.446581 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.446654 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.446940 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.447026 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 15:08:15.458683 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.456423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.9.12.252:34449)
0928 15:08:15.456636 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.456640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.457114 (+ 474us) server_negotiation.cc:407] Connection header received
0928 15:08:15.457323 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.457330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.457425 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.457549 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.458406 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55724: BlockingRecv error: recv got EOF from 127.0.0.1:55724 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 15:08:15.532054 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.526507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.9.12.253:44729)
0928 15:08:15.526701 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.526708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.527080 (+ 372us) server_negotiation.cc:407] Connection header received
0928 15:08:15.527313 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.527317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.527391 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.527495 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.527952 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.528947 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.530290 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.530722 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.530860 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.531281 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.531388 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.531491 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.531669 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.531744 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 15:08:15.540030 31900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:15.537853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.9.12.252:34449)
0928 15:08:15.538031 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.538037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.538525 (+ 488us) server_negotiation.cc:407] Connection header received
0928 15:08:15.538679 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.538683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.538773 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.538895 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:15.539777 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55734: BlockingRecv error: recv got EOF from 127.0.0.1:55734 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:08:15.643258 31914 catalog_manager.cc:1259] Loaded cluster ID: 1d02a720df4841aa875d39607199c201
I20260928 15:08:15.643392 31914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: loading cluster ID for follower catalog manager: success
I20260928 15:08:15.645879 31914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: acquiring CA information for follower catalog manager: success
I20260928 15:08:15.647644 31914 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db7008afa7464e3393d26ff661d6b444: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:08:15.676112 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.671607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.9.12.253:44729)
0928 15:08:15.671779 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.671784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.672107 (+ 323us) server_negotiation.cc:407] Connection header received
0928 15:08:15.672320 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.672324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.672401 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.672491 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.672849 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.673530 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.674745 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.675131 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.675217 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.675560 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.675667 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.675728 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.675855 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.675918 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:08:15.687000 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.681388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.9.12.252:34449)
0928 15:08:15.681572 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.681580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.681938 (+ 358us) server_negotiation.cc:407] Connection header received
0928 15:08:15.682139 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.682143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.682295 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.682414 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.682802 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.683519 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.684899 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.685293 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.685435 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.685916 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.686404 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.686561 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.686611 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.686710 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:08:15.700484 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.692935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.9.12.253:44729)
0928 15:08:15.693200 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.693209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.694283 (+ 1074us) server_negotiation.cc:407] Connection header received
0928 15:08:15.694546 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.694551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.694647 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.694709 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.695458 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.696562 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.698641 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.699131 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.699256 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.699667 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.699783 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.699881 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.700079 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.700154 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:08:15.700846 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.694417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.9.12.252:34449)
0928 15:08:15.694612 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.694617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.694878 (+ 261us) server_negotiation.cc:407] Connection header received
0928 15:08:15.695043 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.695050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.695142 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.695232 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.695925 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.696938 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.698937 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.699490 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.699558 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.700063 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.700164 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.700251 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.700479 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.700549 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:08:15.701704 32068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:15.692800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.9.12.254:39883)
0928 15:08:15.693632 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:15.693640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:15.693942 (+ 302us) server_negotiation.cc:407] Connection header received
0928 15:08:15.694742 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:15.694748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:15.694976 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:15.695164 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:15.695874 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.696679 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.698269 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:15.698741 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:15.698855 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:15.699744 (+ 889us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:15.700184 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:15.700302 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:15.701309 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0928 15:08:15.701409 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":544,"threads_started":1}
I20260928 15:08:15.706074 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31682
I20260928 15:08:15.707129 31744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:15.816569 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31682
I20260928 15:08:15.837636 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31756
I20260928 15:08:15.838685 31818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:15.945060 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31756
I20260928 15:08:15.964906 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 31830
I20260928 15:08:15.965981 31892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:16.071051 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 31830
2026-09-28T15:08:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4286 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:08:16 dist-test-slave-dmhs krb5kdc[32076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:08:16 dist-test-slave-dmhs krb5kdc[32076](info): set up 2 sockets
Sep 28 15:08:16 dist-test-slave-dmhs krb5kdc[32076](info): commencing operation
krb5kdc: starting...
W20260928 15:08:17.181396 9267 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/test-user.keytab.
Sep 28 15:08:17 dist-test-slave-dmhs krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.12.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
Entry for principal kudu/127.9.12.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab.
I20260928 15:08:17.439818 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:35977
--webserver_interface=127.9.12.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:35977,127.9.12.253:46171,127.9.12.252:40997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:08:17.763963 32089 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:08:17.764350 32089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:17.764478 32089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:17.764612 32089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:17.771875 32089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:17.771993 32089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:17.772125 32089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:17.772212 32089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:17.772341 32089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:17.782038 32089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:35977,127.9.12.253:46171,127.9.12.252:40997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.254.keytab
--principal=kudu/127.9.12.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.254:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.12.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:17.783383 32089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:17.784758 32089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:17.796556 32094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:17.796497 32095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:17.796543 32097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:17.797742 32089 server_base.cc:1034] running on GCE node
I20260928 15:08:17.798516 32089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:17.799829 32089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:17.801072 32089 hybrid_clock.cc:648] HybridClock initialized: now 1790608097801009 us; error 37 us; skew 500 ppm
I20260928 15:08:17.801403 32089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:17 dist-test-slave-dmhs krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:17.808696 32089 init.cc:377] Logged in from keytab as kudu/127.9.12.254@KRBTEST.COM (short username kudu)
I20260928 15:08:17.810096 32089 webserver.cc:460] Webserver started at http://127.9.12.254:40511/ using document root <none> and password file <none>
I20260928 15:08:17.810789 32089 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:17.810925 32089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:17.811250 32089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:17.814291 32089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data/instance:
uuid: "2bbb75126b0043308c366ac7c02e36d3"
format_stamp: "Formatted at 2026-09-28 15:08:17 on dist-test-slave-dmhs"
server_key: "c267722b9d26a8f2f3e82bf95b6f5694"
server_key_iv: "1541ddbacc7fcbccf854f097d51778bb"
server_key_version: "encryptionkey@0"
I20260928 15:08:17.815158 32089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal/instance:
uuid: "2bbb75126b0043308c366ac7c02e36d3"
format_stamp: "Formatted at 2026-09-28 15:08:17 on dist-test-slave-dmhs"
server_key: "c267722b9d26a8f2f3e82bf95b6f5694"
server_key_iv: "1541ddbacc7fcbccf854f097d51778bb"
server_key_version: "encryptionkey@0"
I20260928 15:08:17.820138 32089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:08:17.823464 32104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:17.824792 32089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:08:17.824980 32089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal
uuid: "2bbb75126b0043308c366ac7c02e36d3"
format_stamp: "Formatted at 2026-09-28 15:08:17 on dist-test-slave-dmhs"
server_key: "c267722b9d26a8f2f3e82bf95b6f5694"
server_key_iv: "1541ddbacc7fcbccf854f097d51778bb"
server_key_version: "encryptionkey@0"
I20260928 15:08:17.825248 32089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:17.848347 32089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:17.852228 32089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:17.852519 32089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:17.870998 32089 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.254:35977
I20260928 15:08:17.871006 32155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.254:35977 every 8 connection(s)
I20260928 15:08:17.872438 32089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/data/info.pb
I20260928 15:08:17.876149 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 32089
I20260928 15:08:17.876207 32156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:17.876384 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-0/wal/instance
I20260928 15:08:17.876852 9267 external_mini_cluster.cc:1428] Setting key e84d5801b70c82d8d9c201d371457cbe
I20260928 15:08:17.882827 32156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:17 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.253@KRBTEST.COM" created.
I20260928 15:08:17.907302 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:17.886219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.9.12.254:35977)
0928 15:08:17.887155 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:17.887171 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:17.887712 (+ 541us) server_negotiation.cc:407] Connection header received
0928 15:08:17.889757 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:17.889768 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:17.890072 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:17.890274 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:17.891392 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:17.893372 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:17.894719 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:17.895768 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:17.895970 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:17.899441 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:17.899454 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:17.899472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:17.899531 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:17.903018 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:17.903795 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:17.903801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:17.903803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:17.903945 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:17.904496 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:17.904499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:17.904501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:17.905035 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:17.905238 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:17.905887 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 15:08:17.906159 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":664,"thread_start_us":434,"threads_started":1}
I20260928 15:08:17.907344 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:17.886219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:35977 (local address 127.0.0.1:35064)
0928 15:08:17.887528 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:17.887577 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:17.889396 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:17.890304 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:17.890309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:17.890844 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:17.891175 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:17.891193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:17.893525 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:17.893528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:17.894563 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:17.894574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:17.895990 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:17.895995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:17.896106 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:17.896843 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:17.896865 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:17.899201 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:17.903220 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:17.903236 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:17.903250 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:17.903672 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:17.904045 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:17.904048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:17.904050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:17.904396 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:17.905189 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:17.905195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:17.905477 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 15:08:17.905766 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 15:08:17.906333 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":33,"thread_start_us":822,"threads_started":1}
I20260928 15:08:17.910593 32156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:17.911763 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:17.911252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37586)
0928 15:08:17.911458 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:17.911599 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 15:08:17.912051 32108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
I20260928 15:08:17.913688 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:17.913225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37588)
0928 15:08:17.913415 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:17.913505 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:08:17.913889 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:46171: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab.
Entry for principal kudu/127.9.12.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab.
I20260928 15:08:17.935824 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:46171
--webserver_interface=127.9.12.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab
--principal=kudu/127.9.12.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:35977,127.9.12.253:46171,127.9.12.252:40997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:08:17.963263 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } attempt: 1
I20260928 15:08:17.964507 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:17.964025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37594)
0928 15:08:17.964223 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:17.964366 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:08:17.965540 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:17.965131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37602)
0928 15:08:17.965312 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:17.965393 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 15:08:17.965740 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:46171: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
I20260928 15:08:18.034015 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } attempt: 2
I20260928 15:08:18.035176 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.034715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37604)
0928 15:08:18.034945 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.035032 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:08:18.036201 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.035844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37612)
0928 15:08:18.035998 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.036073 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:08:18.036476 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:46171: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
I20260928 15:08:18.127763 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } attempt: 3
I20260928 15:08:18.128947 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.128486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37618)
0928 15:08:18.128691 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.128789 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:08:18.129999 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.129626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37620)
0928 15:08:18.129788 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.129867 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:08:18.130241 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:46171: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
W20260928 15:08:18.258002 32163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:08:18.258311 32163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:18.258380 32163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:18.258457 32163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:18.265498 32163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:18.265583 32163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:18.265641 32163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:18.265690 32163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:18.265749 32163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:18.273506 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } attempt: 4
I20260928 15:08:18.274745 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.274244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37622)
0928 15:08:18.274530 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.274618 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 15:08:18.275676 32163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:35977,127.9.12.253:46171,127.9.12.252:40997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.253.keytab
--principal=kudu/127.9.12.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.253:46171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.12.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:18.275854 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.275462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37634)
0928 15:08:18.275605 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.275686 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:08:18.276096 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.253:46171: Network error: Client connection negotiation failed: client connection to 127.9.12.253:46171: connect: Connection refused (error 111)
I20260928 15:08:18.277068 32163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:18.278367 32163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:08:18.290218 32168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:18.290263 32169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:18.290413 32171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:18.290992 32163 server_base.cc:1034] running on GCE node
I20260928 15:08:18.291706 32163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:18.293154 32163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:18.294395 32163 hybrid_clock.cc:648] HybridClock initialized: now 1790608098294333 us; error 48 us; skew 500 ppm
I20260928 15:08:18.294718 32163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:18.297818 32163 init.cc:377] Logged in from keytab as kudu/127.9.12.253@KRBTEST.COM (short username kudu)
I20260928 15:08:18.299189 32163 webserver.cc:460] Webserver started at http://127.9.12.253:42135/ using document root <none> and password file <none>
I20260928 15:08:18.299788 32163 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:18.299882 32163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:18.300128 32163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:18.302963 32163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data/instance:
uuid: "7a550419b6e641b0a89042732a039dcc"
format_stamp: "Formatted at 2026-09-28 15:08:18 on dist-test-slave-dmhs"
server_key: "1f2ad8d43d41b322cb0e64faa94f85be"
server_key_iv: "b3c8af4e6714c38607bbfc13daded242"
server_key_version: "encryptionkey@0"
I20260928 15:08:18.303671 32163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal/instance:
uuid: "7a550419b6e641b0a89042732a039dcc"
format_stamp: "Formatted at 2026-09-28 15:08:18 on dist-test-slave-dmhs"
server_key: "1f2ad8d43d41b322cb0e64faa94f85be"
server_key_iv: "b3c8af4e6714c38607bbfc13daded242"
server_key_version: "encryptionkey@0"
I20260928 15:08:18.308705 32163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:08:18.311923 32178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:18.313264 32163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:08:18.313416 32163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal
uuid: "7a550419b6e641b0a89042732a039dcc"
format_stamp: "Formatted at 2026-09-28 15:08:18 on dist-test-slave-dmhs"
server_key: "1f2ad8d43d41b322cb0e64faa94f85be"
server_key_iv: "b3c8af4e6714c38607bbfc13daded242"
server_key_version: "encryptionkey@0"
I20260928 15:08:18.313589 32163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:18.333537 32163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:18.337100 32163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:18.337342 32163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:18.354666 32163 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.253:46171
I20260928 15:08:18.354749 32229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.253:46171 every 8 connection(s)
I20260928 15:08:18.356000 32163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/data/info.pb
I20260928 15:08:18.359678 32230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:18.360018 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 32163
I20260928 15:08:18.360204 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-1/wal/instance
I20260928 15:08:18.360680 9267 external_mini_cluster.cc:1428] Setting key 3500f2fe176b9908e1244ed08365af94
I20260928 15:08:18.366107 32230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.12.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.12.252@KRBTEST.COM" created.
I20260928 15:08:18.385632 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.368719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.9.12.254:35977)
0928 15:08:18.368927 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.368933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.370250 (+ 1317us) server_negotiation.cc:407] Connection header received
0928 15:08:18.372004 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.372011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.372106 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.372216 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.373692 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.374378 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.375869 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.376368 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.376508 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.379835 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.379840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.379846 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.379883 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.382495 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.383251 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.383258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.383262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.383393 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.383837 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.383843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.383846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.384126 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.384301 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.385057 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.385282 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:08:18.386620 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.369287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:35977 (local address 127.0.0.1:35070)
0928 15:08:18.370090 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.370126 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:18.371737 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:18.372459 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:18.372466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:18.372991 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:18.373515 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.373531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.374568 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.374571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.375692 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.375704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.376599 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.376606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.376831 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.377503 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:18.377524 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:18.379637 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:18.382661 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.382669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.382685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.383131 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.383524 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.383529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.383533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.383712 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.384305 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:18.384317 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:18.384674 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 15:08:18.384953 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 15:08:18.385462 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":368,"threads_started":1}
I20260928 15:08:18.388325 32230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:08:18.403160 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.389408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.9.12.253:46171)
0928 15:08:18.390006 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.390012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.390034 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:08:18.390074 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.390079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.390235 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.390388 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.391154 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.392443 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.393573 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.394107 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.394236 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.397641 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.397652 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.397663 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.397691 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.400373 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.401117 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.401122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.401125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.401227 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.401634 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.401637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.401639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.401950 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.402110 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.402592 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.402881 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":458,"thread_start_us":322,"threads_started":1}
I20260928 15:08:18.403258 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.389188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37642)
0928 15:08:18.389382 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.389399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:18.389491 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:18.390372 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:18.390376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:18.390812 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:18.390957 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.390965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.392591 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.392598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.393424 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.393434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.394243 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.394248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.394476 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.395040 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:18.395056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:18.397444 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:18.400491 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.400498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.400504 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.401000 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.401334 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.401341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.401347 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.401528 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.402101 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:18.402109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:18.402288 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 15:08:18.402453 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:08:18.402768 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":42}
I20260928 15:08:18.405681 32230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:18.406701 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.406264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59722)
0928 15:08:18.406440 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.406557 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:08:18.406947 32182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.408566 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.408208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59730)
0928 15:08:18.408380 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.408465 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 15:08:18.408771 32230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.12.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab.
Entry for principal kudu/127.9.12.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab.
I20260928 15:08:18.419623 9267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
/tmp/dist-test-task7VZsZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:40997
--webserver_interface=127.9.12.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab
--principal=kudu/127.9.12.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.12.212:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.12.254:35977,127.9.12.253:46171,127.9.12.252:40997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:08:18.458070 32230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 1
I20260928 15:08:18.459257 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.458738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59746)
0928 15:08:18.458928 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.459044 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 15:08:18.461331 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.460856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59752)
0928 15:08:18.461048 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.461139 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 15:08:18.461598 32230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.529799 32230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 2
I20260928 15:08:18.530923 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.530467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59758)
0928 15:08:18.530676 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.530772 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 15:08:18.531872 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.531554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59762)
0928 15:08:18.531697 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.531768 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:08:18.532078 32230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.575448 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } attempt: 5
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:08:18.588677 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.576367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.9.12.253:46171)
0928 15:08:18.576561 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.576568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.576590 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:08:18.576816 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.576820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.576881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.576987 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.577772 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.578454 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.579646 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.580098 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.580231 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.583605 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.583609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.583612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.583641 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.586136 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.586745 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.586751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.586754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.586902 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.587283 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.587288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.587292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.587570 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.587726 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.588118 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.588341 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:08:18.588677 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.576242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37644)
0928 15:08:18.576489 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.576512 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:18.576653 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:18.577050 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:18.577058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:18.577451 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:18.577616 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.577624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.578603 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.578611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.579484 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.579496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.580244 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.580250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.580480 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.581167 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:18.581184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:18.583403 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:18.586267 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.586272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.586276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.586630 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.587022 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.587027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.587031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.587173 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.587690 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:18.587695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:18.587853 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:08:18.588021 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 15:08:18.588330 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 15:08:18.589888 32156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:08:18.590893 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.590532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59774)
0928 15:08:18.590677 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.590745 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:08:18.591773 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.591470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59782)
0928 15:08:18.591604 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.591675 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:08:18.592022 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.623293 32230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 3
I20260928 15:08:18.624483 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.623945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59784)
0928 15:08:18.624176 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.624318 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:08:18.625504 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.625113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59794)
0928 15:08:18.625321 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.625399 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260928 15:08:18.625730 32230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.643265 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 1
I20260928 15:08:18.644532 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.644018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59806)
0928 15:08:18.644224 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.644366 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:08:18.645489 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.645213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59816)
0928 15:08:18.645336 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.645399 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 15:08:18.645740 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.713979 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 2
I20260928 15:08:18.715234 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.714779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59826)
0928 15:08:18.714972 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.715086 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:08:18.716431 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.716043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59840)
0928 15:08:18.716186 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.716247 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:08:18.716714 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
W20260928 15:08:18.750351 32237 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:08:18.750669 32237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:08:18.750733 32237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:08:18.750811 32237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:08:18.758134 32237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:08:18.758220 32237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:08:18.758276 32237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:08:18.758323 32237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:08:18.758384 32237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:08:18.768116 32237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.12.212:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.12.254:35977,127.9.12.253:46171,127.9.12.252:40997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/krb5kdc/kudu_127.9.12.252.keytab
--principal=kudu/127.9.12.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.12.252:40997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.12.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:00:13 UTC on ccycloud.quasar-sekvba.root.comops.site
build id 4677370
ASAN enabled
I20260928 15:08:18.768949 32230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 4
I20260928 15:08:18.769518 32237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:08:18.770112 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.769659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59844)
0928 15:08:18.769874 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.769961 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:08:18.771003 32237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:08:18.771123 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.770780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59848)
0928 15:08:18.770918 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.770991 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:08:18.771341 32230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
W20260928 15:08:18.782946 32243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:18.782878 32245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:08:18.783597 32242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:08:18.789132 32237 server_base.cc:1034] running on GCE node
I20260928 15:08:18.790047 32237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:08:18.791559 32237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:08:18.792798 32237 hybrid_clock.cc:648] HybridClock initialized: now 1790608098792731 us; error 48 us; skew 500 ppm
I20260928 15:08:18.793159 32237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:08:18.796126 32237 init.cc:377] Logged in from keytab as kudu/127.9.12.252@KRBTEST.COM (short username kudu)
I20260928 15:08:18.797562 32237 webserver.cc:460] Webserver started at http://127.9.12.252:35759/ using document root <none> and password file <none>
I20260928 15:08:18.798262 32237 fs_manager.cc:359] Metadata directory not provided
I20260928 15:08:18.798396 32237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:08:18.798710 32237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:08:18.801643 32237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data/instance:
uuid: "25f5758bd5f04ec89aeea7c64b12bd14"
format_stamp: "Formatted at 2026-09-28 15:08:18 on dist-test-slave-dmhs"
server_key: "3965a9689d6d079d193d709b8305d568"
server_key_iv: "7fff813a88e48893b7b9d30fc2c7c211"
server_key_version: "encryptionkey@0"
I20260928 15:08:18.802402 32237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal/instance:
uuid: "25f5758bd5f04ec89aeea7c64b12bd14"
format_stamp: "Formatted at 2026-09-28 15:08:18 on dist-test-slave-dmhs"
server_key: "3965a9689d6d079d193d709b8305d568"
server_key_iv: "7fff813a88e48893b7b9d30fc2c7c211"
server_key_version: "encryptionkey@0"
I20260928 15:08:18.807361 32237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:08:18.810719 32252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:08:18.812036 32237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:08:18.812222 32237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data,/tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal
uuid: "25f5758bd5f04ec89aeea7c64b12bd14"
format_stamp: "Formatted at 2026-09-28 15:08:18 on dist-test-slave-dmhs"
server_key: "3965a9689d6d079d193d709b8305d568"
server_key_iv: "7fff813a88e48893b7b9d30fc2c7c211"
server_key_version: "encryptionkey@0"
I20260928 15:08:18.812492 32237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:08:18.822935 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 3
I20260928 15:08:18.824059 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.823563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59860)
0928 15:08:18.823779 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.823877 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:08:18.825151 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.824785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59872)
0928 15:08:18.824951 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.825038 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:08:18.825346 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.12.252:40997: Network error: Client connection negotiation failed: client connection to 127.9.12.252:40997: connect: Connection refused (error 111)
I20260928 15:08:18.837847 32237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:08:18.841753 32237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:08:18.841969 32237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:08:18.860453 32237 rpc_server.cc:307] RPC server started. Bound to: 127.9.12.252:40997
I20260928 15:08:18.860576 32303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.12.252:40997 every 8 connection(s)
I20260928 15:08:18.861768 32237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/data/info.pb
I20260928 15:08:18.864339 9267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu as pid 32237
I20260928 15:08:18.864518 9267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7VZsZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607741729566-9267-0/minicluster-data/master-2/wal/instance
I20260928 15:08:18.864975 9267 external_mini_cluster.cc:1428] Setting key 134f8342b7472db733175ab1a92fff42
I20260928 15:08:18.865487 32304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:08:18.875463 32304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
I20260928 15:08:18.886870 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.866698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.9.12.252:40997)
0928 15:08:18.867474 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.867485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.867523 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:08:18.869040 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.869061 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.869444 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.869843 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.871327 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.872641 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.873901 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.874538 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.874733 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.878983 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.878999 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.879017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.879058 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.882785 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.883434 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.883442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.883449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.883652 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.884073 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.884076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.884078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.884722 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.885023 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.885554 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.885949 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":345,"threads_started":1}
I20260928 15:08:18.893795 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.878268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.9.12.254:35977)
0928 15:08:18.878473 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.878481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.879169 (+ 688us) server_negotiation.cc:407] Connection header received
0928 15:08:18.879450 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.879456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.879541 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.879639 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.881130 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.881827 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.883595 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.884248 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.884403 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.887719 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.887726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.887731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.887778 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.891165 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.891824 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.891828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.891830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.891992 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.892388 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.892393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.892399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.892669 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.892833 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.893246 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.893439 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:08:18.893844 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.878197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.254:35977 (local address 127.0.0.1:35076)
0928 15:08:18.879037 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.879078 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:18.879298 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:18.879716 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:18.879725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:18.880592 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:18.880971 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.880988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.881994 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.882000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.883454 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.883467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.884468 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.884474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.884634 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.885300 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:18.885322 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:18.887534 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:18.891283 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.891295 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.891309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.891660 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.892105 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.892110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.892114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.892250 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.892793 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:18.892800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:18.892976 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 15:08:18.893145 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 15:08:18.893439 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":443,"threads_started":1}
I20260928 15:08:18.895139 32304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.254@KRBTEST.COM
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
I20260928 15:08:18.904677 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.889798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.9.12.254:35977)
0928 15:08:18.890436 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.890441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.890464 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:08:18.890509 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.890514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.890581 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.890689 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.891741 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.892668 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.893875 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.894434 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.894564 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.899092 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.899100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.899104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.899137 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.901890 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.902504 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.902512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.902517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.902698 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.903055 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.903061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.903066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.903440 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.903704 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.903996 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.904206 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260928 15:08:18.908679 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.895854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.253:46171 (local address 127.0.0.1:37654)
0928 15:08:18.896034 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.896053 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:18.896151 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:18.896529 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:18.896535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:18.896987 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:18.897184 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.897195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.898134 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.898139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.899001 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.899012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.899989 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.899994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.900117 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.900625 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:18.900641 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:18.902562 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:18.906177 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.906180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.906182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.906665 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.907121 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.907127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.907130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.907267 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.907781 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:18.907785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:18.907926 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 15:08:18.908058 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:08:18.908335 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:08:18.908809 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.895895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.9.12.253:46171)
0928 15:08:18.896078 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.896082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.896099 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:08:18.896312 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.896317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.896395 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.896471 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.897320 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.897996 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.899128 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.899806 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.899999 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.902757 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.902765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.902773 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.902814 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.906053 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.906859 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.906863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.906865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.907003 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.907380 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.907386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.907389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.907664 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.907813 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.908148 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.908390 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:08:18.909689 32304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.12.253@KRBTEST.COM
Sep 28 15:08:18 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.252@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:08:18.922075 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.910291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59882)
0928 15:08:18.910453 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:18.910469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:18.910553 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:18.910854 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:18.910857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:18.911189 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:18.911342 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.911352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.912537 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.912545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.913358 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:18.913368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.914035 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.914040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:18.914234 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.914814 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:18.914838 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:18.917199 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:18.919811 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.919816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.919820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.920155 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.920526 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:18.920531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:18.920534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:18.920655 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:18.921110 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:18.921115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:18.921246 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 15:08:18.921383 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 15:08:18.921670 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:08:18.922155 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.910461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.9.12.252:40997)
0928 15:08:18.910612 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.910617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.910634 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:08:18.910674 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.910678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.910739 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.910834 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.911491 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.912424 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.913490 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.913890 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.914019 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.917346 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.917351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.917356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.917392 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.919677 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.920287 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.920295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.920307 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.920425 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.920751 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.920758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.920762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.921008 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.921144 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.921459 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.921822 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":53}
I20260928 15:08:18.922281 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:18.906448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.9.12.253:46171)
0928 15:08:18.907031 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:18.907035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:18.907054 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:08:18.907091 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:18.907094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:18.907141 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:18.907255 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:18.908498 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.909611 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.911161 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:18.911713 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:18.911795 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:18.916284 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:18.916293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:18.916297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:18.916333 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:18.919523 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.920196 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.920201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.920205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.920397 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:18.920766 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:18.920771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:18.920774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:18.921035 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:18.921277 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:18.921553 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 15:08:18.921754 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":261,"threads_started":1}
I20260928 15:08:18.926849 9267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:08:18.927414 32304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: Bootstrap starting.
I20260928 15:08:18.931010 32304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:18.932004 32304 log.cc:824] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:18.929517 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2bbb75126b0043308c366ac7c02e36d3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090
I20260928 15:08:18.934677 32304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: No bootstrap required, opened a new log
I20260928 15:08:18.941349 32304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:18.941655 32304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f5758bd5f04ec89aeea7c64b12bd14, State: Initialized, Role: FOLLOWER
I20260928 15:08:18.942057 32304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:18.942986 32312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:18.943370 32312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:18.943840 32304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:08:18.952888 32323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:08:18.953011 32323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:08:18.950835 32237 master_runner.cc:186] Error getting master registration for 127.9.12.254:35977: OK, instance_id { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" instance_seqno: 1790608097865629 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:08:18.961849 32237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:08:19.002561 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 4
Sep 28 15:08:19 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.254@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:08:19.017756 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.003362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59890)
0928 15:08:19.003564 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:19.003583 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:19.003716 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:19.004105 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:19.004112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:19.004548 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:19.004697 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:19.004705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.005699 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.005706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:19.006616 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:19.006627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.007428 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.007434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:19.007648 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.008303 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:19.008329 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:19.010822 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:19.014883 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:19.014890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:19.014896 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:19.015393 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:19.015769 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:19.015773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:19.015777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:19.015966 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:19.016683 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:19.016688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:19.016900 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 15:08:19.017056 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 15:08:19.017347 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:08:19.017765 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.003567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.9.12.252:40997)
0928 15:08:19.003794 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.003799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.003820 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:08:19.003906 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.003909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.003963 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.004067 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.004850 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.005555 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.006761 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.007281 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.007415 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.011034 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:19.011041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:19.011048 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:19.011089 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:19.014740 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.015508 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.015513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.015517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.015652 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.016076 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.016080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.016083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.016565 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:19.016719 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:19.017177 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 15:08:19.017400 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:08:19.023985 32156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3: Bootstrap starting.
I20260928 15:08:19.028106 32156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:19.029553 32156 log.cc:824] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:08:19.032948 32156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3: No bootstrap required, opened a new log
I20260928 15:08:19.040483 32156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:19.040861 32156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbb75126b0043308c366ac7c02e36d3, State: Initialized, Role: FOLLOWER
I20260928 15:08:19.041333 32156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:19.042320 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.042726 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:19.043550 32156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:19.043612 32131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:08:19.043829 32131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:19.046803 32131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:19.048126 32131 leader_election.cc:288] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [CANDIDATE]: Term 1 election: Requested vote from peers 7a550419b6e641b0a89042732a039dcc (127.9.12.253:46171), 25f5758bd5f04ec89aeea7c64b12bd14 (127.9.12.252:40997)
I20260928 15:08:19.049780 32279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbb75126b0043308c366ac7c02e36d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25f5758bd5f04ec89aeea7c64b12bd14"
I20260928 15:08:19.050379 32279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:19.055058 32279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbb75126b0043308c366ac7c02e36d3 in term 1.
I20260928 15:08:19.056054 32108 leader_election.cc:302] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f5758bd5f04ec89aeea7c64b12bd14, 2bbb75126b0043308c366ac7c02e36d3; no voters:
I20260928 15:08:19.056605 32326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:08:19.068074 32326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2bbb75126b0043308c366ac7c02e36d3, State: Running, Role: LEADER
I20260928 15:08:19.069460 32326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:19.070549 32230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } attempt: 5
Sep 28 15:08:19 dist-test-slave-dmhs krb5kdc[32076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.12.253@KRBTEST.COM for kudu/127.9.12.252@KRBTEST.COM
I20260928 15:08:19.083464 32331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbb75126b0043308c366ac7c02e36d3. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.084442 32331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:19.088060 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.071134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.12.252:40997 (local address 127.0.0.1:59892)
0928 15:08:19.071635 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 15:08:19.071655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:08:19.071774 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:08:19.072459 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:08:19.072465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:08:19.072986 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:08:19.073205 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:19.073216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.074700 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.074705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:19.076144 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:08:19.076160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.077329 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.077335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:08:19.077572 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.078261 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:08:19.078286 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:08:19.081046 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:08:19.085156 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:19.085161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:19.085165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:19.085666 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:19.086091 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:08:19.086096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:08:19.086100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:08:19.086263 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:08:19.087019 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:08:19.087025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:08:19.087174 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 15:08:19.087339 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:08:19.087624 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260928 15:08:19.089128 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.071429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.9.12.252:40997)
0928 15:08:19.072004 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.072011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.072027 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:08:19.072082 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.072087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.072159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.072246 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.073373 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.074535 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.076334 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.076995 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.077118 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.081237 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:19.081243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:19.081248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:19.081289 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:19.085020 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.085817 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.085822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.085826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.085966 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.086399 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.086404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.086408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.086741 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:19.086943 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:19.088576 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0928 15:08:19.088784 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455}
W20260928 15:08:19.093950 32258 tablet.cc:2369] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:08:19.095360 32230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: Bootstrap starting.
I20260928 15:08:19.095494 32336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:08:19.099316 32230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: Neither blocks nor log segments found. Creating new log.
I20260928 15:08:19.100899 32230 log.cc:824] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: Log is configured to *not* fsync() on all Append() calls
W20260928 15:08:19.105257 32110 tablet.cc:2369] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:08:19.118678 32230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: No bootstrap required, opened a new log
I20260928 15:08:19.119266 32336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:08:19.145170 32230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:19.145736 32230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a550419b6e641b0a89042732a039dcc, State: Initialized, Role: FOLLOWER
I20260928 15:08:19.146355 32230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } }
I20260928 15:08:19.150478 32348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.150977 32348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:19.158208 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.113238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.9.12.254:35977)
0928 15:08:19.113435 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.113442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.124273 (+ 10831us) server_negotiation.cc:407] Connection header received
0928 15:08:19.124759 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.124764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.124842 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.124917 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.127254 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.128394 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.130575 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.131284 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.131400 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.134936 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:19.134945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:19.134951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:19.134993 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:19.139901 (+ 4908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.141207 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.141215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.141244 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.141545 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.142258 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.142267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.142272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.143123 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:19.156221 (+ 13098us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:19.156623 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 15:08:19.157199 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:08:19.158526 32230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:08:19.158740 32205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bbb75126b0043308c366ac7c02e36d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a550419b6e641b0a89042732a039dcc"
I20260928 15:08:19.159332 32205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:08:19.163975 32205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbb75126b0043308c366ac7c02e36d3 in term 1.
I20260928 15:08:19.196621 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.173200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.9.12.252:40997)
0928 15:08:19.173464 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.173473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.174150 (+ 677us) server_negotiation.cc:407] Connection header received
0928 15:08:19.174465 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.174474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.174590 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.174674 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.177137 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.178592 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.181340 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.182092 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.182173 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.187101 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:19.187107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:19.187112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:19.187149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:19.191253 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.193430 (+ 2177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.193439 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.193446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.193623 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.194164 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.194172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.194178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.194592 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:19.194814 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:19.195966 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0928 15:08:19.196194 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:08:19.199414 32279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [term 1 FOLLOWER]: Refusing update from remote peer 2bbb75126b0043308c366ac7c02e36d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:08:19.199406 32205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [term 1 FOLLOWER]: Refusing update from remote peer 2bbb75126b0043308c366ac7c02e36d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:08:19.200944 32326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:08:19.201830 32326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:08:19.202754 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:19.119604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.9.12.253:46171)
0928 15:08:19.119850 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.119867 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.145419 (+ 25552us) server_negotiation.cc:407] Connection header received
0928 15:08:19.145509 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.145517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.145706 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.145824 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.150010 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.151567 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.166270 (+ 14703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:19.167236 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:19.167361 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:19.180589 (+ 13228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:08:19.180599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:08:19.180606 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:08:19.180659 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:08:19.184783 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.195272 (+ 10489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.195279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.195283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.195434 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:08:19.200388 (+ 4954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:08:19.200407 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:08:19.200413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:08:19.200927 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:08:19.201229 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:19.201952 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 15:08:19.202268 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:08:19.239935 32348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbb75126b0043308c366ac7c02e36d3. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.240384 32348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:19.244904 32346 mvcc.cc:205] Tried to move back new op lower bound from 7334330774268489728 to 7334330773804208128. Current Snapshot: MvccSnapshot[applied={T|T < 7334330774268489728}]
I20260928 15:08:19.252841 32357 mvcc.cc:205] Tried to move back new op lower bound from 7334330774268489728 to 7334330773804208128. Current Snapshot: MvccSnapshot[applied={T|T < 7334330774268489728}]
I20260928 15:08:19.258565 32312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbb75126b0043308c366ac7c02e36d3. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.258960 32312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:19.264014 32312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.264252 32312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:19.264940 32356 mvcc.cc:205] Tried to move back new op lower bound from 7334330774268489728 to 7334330773804208128. Current Snapshot: MvccSnapshot[applied={T|T < 7334330774268489728}]
I20260928 15:08:19.270588 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.271024 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:19.271333 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.271589 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:08:19.274575 32348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bbb75126b0043308c366ac7c02e36d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbb75126b0043308c366ac7c02e36d3" member_type: VOTER last_known_addr { host: "127.9.12.254" port: 35977 } } peers { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" member_type: VOTER last_known_addr { host: "127.9.12.253" port: 46171 } } peers { permanent_uuid: "25f5758bd5f04ec89aeea7c64b12bd14" member_type: VOTER last_known_addr { host: "127.9.12.252" port: 40997 } } }
I20260928 15:08:19.274842 32348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:08:19.289026 32336 catalog_manager.cc:1347] Generated new cluster ID: 880624a8fdb14252a0d0bb0b85b24854
I20260928 15:08:19.289175 32336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:08:19.295105 32365 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:08:19.298935 32089 master_runner.cc:186] Error getting master registration for 127.9.12.253:46171: OK, instance_id { permanent_uuid: "7a550419b6e641b0a89042732a039dcc" instance_seqno: 1790608098349527 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:08:19.309480 32089 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:08:19.344746 32336 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:08:19.347502 32368 catalog_manager.cc:1259] Loaded cluster ID: 880624a8fdb14252a0d0bb0b85b24854
I20260928 15:08:19.347604 32368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: loading cluster ID for follower catalog manager: success
I20260928 15:08:19.351377 32368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: acquiring CA information for follower catalog manager: success
W20260928 15:08:19.352762 32368 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260928 15:08:19.352805 32336 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:08:19.377105 32336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bbb75126b0043308c366ac7c02e36d3: Generated new TSK 0
I20260928 15:08:19.379745 32336 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:08:19.501348 32309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.483950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.9.12.254:35977)
0928 15:08:19.484132 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.484138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.497562 (+ 13424us) server_negotiation.cc:407] Connection header received
0928 15:08:19.499652 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.499660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.499745 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.499824 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.501066 (+ 1242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35104: BlockingRecv error: recv got EOF from 127.0.0.1:35104 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 15:08:19.503254 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.492448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.9.12.253:46171)
0928 15:08:19.492630 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.492636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.498111 (+ 5475us) server_negotiation.cc:407] Connection header received
0928 15:08:19.501716 (+ 3605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.501722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.501809 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.501896 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.502977 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37694: BlockingRecv error: recv got EOF from 127.0.0.1:37694 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:08:19.505681 32307 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.498625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.9.12.252:40997)
0928 15:08:19.498794 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.498799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.503806 (+ 5007us) server_negotiation.cc:407] Connection header received
0928 15:08:19.504390 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.504396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.504475 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.504560 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.505463 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59912: BlockingRecv error: recv got EOF from 127.0.0.1:59912 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 15:08:19.573194 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.557977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.9.12.253:46171)
0928 15:08:19.558168 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.558175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.570914 (+ 12739us) server_negotiation.cc:407] Connection header received
0928 15:08:19.571473 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.571479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.571629 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.571708 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.572946 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37702: BlockingRecv error: recv got EOF from 127.0.0.1:37702 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 15:08:19.610998 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.601400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.9.12.253:46171)
0928 15:08:19.601588 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.601594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.608815 (+ 7221us) server_negotiation.cc:407] Connection header received
0928 15:08:19.609318 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.609336 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.609467 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.609542 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.610730 (+ 1188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37710: BlockingRecv error: recv got EOF from 127.0.0.1:37710 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:08:19.628744 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.626486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.9.12.253:46171)
0928 15:08:19.626662 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.626668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.627212 (+ 544us) server_negotiation.cc:407] Connection header received
0928 15:08:19.627431 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.627437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.627541 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.627654 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.628505 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37718: BlockingRecv error: recv got EOF from 127.0.0.1:37718 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:08:19.644330 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.642088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.9.12.253:46171)
0928 15:08:19.642315 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.642321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.642826 (+ 505us) server_negotiation.cc:407] Connection header received
0928 15:08:19.643028 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.643035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.643134 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.643252 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.644049 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37722: BlockingRecv error: recv got EOF from 127.0.0.1:37722 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260928 15:08:19.657603 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.655401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.9.12.253:46171)
0928 15:08:19.655565 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.655571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.656130 (+ 559us) server_negotiation.cc:407] Connection header received
0928 15:08:19.656342 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.656348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.656476 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.656584 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.657370 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37724: BlockingRecv error: recv got EOF from 127.0.0.1:37724 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 15:08:19.679315 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.676967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.9.12.253:46171)
0928 15:08:19.677208 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.677213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.677744 (+ 531us) server_negotiation.cc:407] Connection header received
0928 15:08:19.677952 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.677964 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.678079 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.678199 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.679036 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37728: BlockingRecv error: recv got EOF from 127.0.0.1:37728 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 15:08:19.720717 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.718291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.9.12.253:46171)
0928 15:08:19.718503 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.718512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.719077 (+ 565us) server_negotiation.cc:407] Connection header received
0928 15:08:19.719305 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.719311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.719407 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.719544 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.720473 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37738: BlockingRecv error: recv got EOF from 127.0.0.1:37738 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 15:08:19.799640 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.797366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.9.12.253:46171)
0928 15:08:19.797545 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.797552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.798099 (+ 547us) server_negotiation.cc:407] Connection header received
0928 15:08:19.798295 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.798301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.798426 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.798536 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.799379 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37754: BlockingRecv error: recv got EOF from 127.0.0.1:37754 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:08:19.932415 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:19.930547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.9.12.253:46171)
0928 15:08:19.930776 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:19.930783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:19.931044 (+ 261us) server_negotiation.cc:407] Connection header received
0928 15:08:19.931297 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:19.931303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:19.931398 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:19.931483 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:19.932158 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37756: BlockingRecv error: recv got EOF from 127.0.0.1:37756 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:08:19.954793 32323 catalog_manager.cc:1259] Loaded cluster ID: 880624a8fdb14252a0d0bb0b85b24854
I20260928 15:08:19.954929 32323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: loading cluster ID for follower catalog manager: success
I20260928 15:08:19.957283 32323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: acquiring CA information for follower catalog manager: success
I20260928 15:08:19.958979 32323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25f5758bd5f04ec89aeea7c64b12bd14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:08:20.195779 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:08:20.194069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.9.12.253:46171)
0928 15:08:20.194294 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:20.194300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:20.194485 (+ 185us) server_negotiation.cc:407] Connection header received
0928 15:08:20.194668 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:20.194672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:20.194748 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:20.194835 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:08:20.195535 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37772: BlockingRecv error: recv got EOF from 127.0.0.1:37772 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:08:20.355345 32368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a550419b6e641b0a89042732a039dcc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:08:20.719393 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:20.713652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.9.12.253:46171)
0928 15:08:20.714138 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:20.714146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:20.714278 (+ 132us) server_negotiation.cc:407] Connection header received
0928 15:08:20.714505 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:20.714509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:20.714661 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:20.714790 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:20.715222 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.715931 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.717397 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.717797 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.717892 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:20.718368 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:20.718892 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:20.719003 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:20.719094 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:08:20.719162 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260928 15:08:20.730635 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:20.724974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.9.12.252:40997)
0928 15:08:20.725491 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:20.725497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:20.725515 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:08:20.725745 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:20.725749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:20.725909 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:20.726075 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:20.726449 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.727151 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.728518 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.728951 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.729083 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:20.729553 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:20.730078 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:20.730226 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:20.730281 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 15:08:20.730368 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260928 15:08:20.743716 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:20.736171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.9.12.253:46171)
0928 15:08:20.736356 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:20.736360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:20.737650 (+ 1290us) server_negotiation.cc:407] Connection header received
0928 15:08:20.737942 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:20.737948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:20.738060 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:20.738147 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:20.738962 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.739699 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.741606 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.742233 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.742362 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:20.742960 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:20.743062 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:20.743161 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:20.743370 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:08:20.743453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:08:20.744738 32447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:20.736107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.9.12.254:35977)
0928 15:08:20.737343 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:20.737349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:20.737382 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:08:20.737495 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:20.737503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:20.737725 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:20.738616 (+ 891us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:20.738793 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.739792 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.741672 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.742274 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.742389 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:20.743363 (+ 974us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:20.743762 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:20.744274 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:20.744421 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 15:08:20.744509 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":976,"threads_started":1}
I20260928 15:08:20.744753 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:08:20.737830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.9.12.252:40997)
0928 15:08:20.737985 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:08:20.737990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:08:20.738620 (+ 630us) server_negotiation.cc:407] Connection header received
0928 15:08:20.738788 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:08:20.738793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:08:20.738896 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:08:20.738984 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:08:20.739588 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.740676 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.742320 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:08:20.742965 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:08:20.743085 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:08:20.743586 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:08:20.743713 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:08:20.743775 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:08:20.744428 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 15:08:20.744504 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:08:20.749075 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 32089
I20260928 15:08:20.750031 32151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:20.858522 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 32089
W20260928 15:08:20.879252 32255 connection.cc:535] server connection from 127.0.0.1:59890 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:08:20.879618 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 32163
I20260928 15:08:20.880733 32225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:20.985054 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 32163
I20260928 15:08:21.005040 9267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu pid 32237
I20260928 15:08:21.006193 32299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:08:21.110517 9267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7VZsZQ/build/asan/bin/kudu with pid 32237
2026-09-28T15:08:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5035 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9321 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (359342 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST